CN113542332B - Customer service video interaction method and equipment based on positioning labels - Google Patents

Abstract

The invention discloses a method and equipment for customer service video interaction based on location marking. The method includes: the customer service system equipment receives the video information transmitted by the terminal side equipment and forwards it to the customer service side equipment for display, wherein the customer service system equipment includes an entity list, a recognition algorithm, and an annotation information library; the customer service system equipment selects a recognition algorithm according to the target entity, According to the recognition algorithm, identify whether there is a target entity from the frame image updated by video information, and if so, identify the position of the target entity to obtain positioning information; the customer service system equipment receives the interaction information, and synchronizes the interaction information and positioning information to the terminal side equipment and customer service side equipment to realize the display of marked information on the terminal side equipment and customer service side equipment. Adopting the customer service video interaction method based on positioning annotation can significantly improve the interaction mode effect of video information annotation.

Description

Customer service video interaction method and device based on positioning annotation

【Technical field】

The present invention relates to the field of Internet applications, in particular to a method and device for customer service video interaction based on location marking.

【Background technique】

With the development of the Internet of Vehicles, the intelligent capabilities of vehicles have gradually improved, and the application of Over the Air (OTA) technology has even improved the speed of related system improvements. However, the problem of faults has also become invisible and complicated. The previous customer service model is gradually difficult to adapt to new service scenarios. It has become a trend to transform the remote expert customer service system through video.

Video customer service is based on the existing voice and online customer service, adding the ability of video access. Compared with the original mode, it adds richer information sharing. It can display the information of the customer service side equipment to the customer service side equipment in real time, so that the customer service can quickly understand On-site situation, to help users answer or solve their practical problems in a timely manner. In addition to voice and text, the current interaction mode between video customer service and terminal-side equipment users also introduces the interaction mode of video information annotation. However, due to objective reasons such as complex interaction scenarios and high requirements on equipment performance, the video There are still many problems in the interactive method of information annotation, and the interactive method of video information annotation is not effective.

【Content of invention】

In view of this, the embodiments of the present invention provide a method and device for customer service video interaction based on location annotation, which are used to solve the poor effect of the current interaction method of video information annotation.

In the first aspect, the embodiment of the present invention provides a customer service video interaction method based on positioning annotation, including the following steps performed by the customer service system equipment:

The customer service system equipment receives the video information transmitted by the terminal side equipment and forwards it to the customer service side equipment for display, wherein the customer service system equipment includes an entity list, a recognition algorithm, and an annotation information library;

The customer service system equipment selects the recognition algorithm according to the target entity, and identifies whether the target entity exists from the frame image updated by the video information according to the recognition algorithm, and if so, performs position recognition on the target entity, Obtaining positioning information, wherein the target entity is selected based on the entity list through a specified instruction input by the customer service side device;

The customer service system equipment receives the interaction information, and synchronizes the interaction information and the positioning information to the terminal side equipment and the customer service side equipment, so as to mark information on the terminal side equipment and the customer service side equipment , wherein the interaction information is acquired through the customer service side device based on the annotation information base.

According to the above aspect and any possible implementation manner, an implementation manner is further provided, before the customer service system equipment selects the identification algorithm according to the target entity, it further includes:

The customer service platform system performs image frame extraction according to the video information at intervals of a preset time interval to obtain updated frame image images of the video information, wherein the preset time interval is 1/3 second.

In the second aspect, the embodiment of the present invention also provides a customer service video interaction method based on positioning annotation, including the following steps performed by the customer service side device:

The customer service side equipment receives the video information forwarded by the customer service system equipment;

The customer service side equipment uses the entity list to select the target entity according to the specified instruction input, wherein the customer service side equipment is in a communication connection with the customer service system equipment, and the customer service system equipment includes the entity list, identification algorithm and labeling information library;

Based on the annotation information base, the customer service side device obtains the interaction information edited by the customer service personnel;

The customer service side equipment sends the interaction information to the customer service system equipment;

The customer service side equipment receives the interaction information and positioning information synchronously sent by the customer service system equipment, wherein the positioning information is that the customer service system equipment selects the identification algorithm according to the target entity, and selects the identification algorithm according to the identification The algorithm performs position recognition on the target entity identified in the frame image updated by the video information;

The customer service side device realizes the display of annotation information based on the interaction information and the positioning information, wherein the interaction information is used for the display of the annotation information of the text type, and the positioning information is used for the display of the graphics type. Display of annotation information.

According to the above-mentioned aspect and any possible implementation manner, an implementation manner is further provided, the customer service side device realizes the display of annotation information based on the interaction information and the positioning information, including:

The customer service side device circles the target entity by using the positioning information and the preset labeling formula, so as to realize the display of labeling information circled by the entity;

The customer service side device generates a click area on the area where the target entity is circled based on the interaction information, and the click area is used to display the interaction information and the information reply area after receiving a click instruction, wherein the The interaction information includes text information and data information, the data information is displayed in the form of a hyperlink, and the information reply area is used to display or reply to the reply information sent by the terminal side device;

The customer service side device realizes displaying the serial numbers of different target entities according to the location information based on the pre-edited serial numbers of the different target entities.

According to the above aspect and any possible implementation, an implementation is further provided, the positioning information includes the center position (x, y) and the area pixel size (a, b), and the customer service side device adopts the The positioning information and the preset labeling formula circle the target entity to realize the display of labeling information circled by the entity, including:

其中，设定所述目标实体的序号显示的显示位置为(X+a/2，Y+b/2)。According to the center position (x, y) and the area pixel size (a, b), the customer service side device calculates the encircled figure with the pixel point set as F(Xn, Yn), and the formula is

Wherein, the display position of the serial number display of the target entity is set as (X+a/2, Y+b/2).

In the third aspect, the embodiment of the present invention also provides a method for customer service video interaction based on location annotation, including the following steps performed by the terminal-side device:

The terminal side equipment transmits the video information to the customer service system equipment, wherein the customer service system equipment includes an entity list, a recognition algorithm, and an annotation information library;

The terminal side device receives the interaction information and positioning information synchronously sent by the customer service system equipment, wherein the interaction information is obtained by the customer service side device based on the annotation information database, and the positioning information is the customer service system The device selects the recognition algorithm according to the target entity, and according to the recognition algorithm, performs position recognition on the target entity identified in the frame image updated by the video information, and obtains the target entity based on the entity list through the The specified instruction input by the customer service side equipment is selected;

The terminal-side device realizes the display of annotation information based on the interaction information and the positioning information, wherein the interaction information is used for the display of the annotation information of the text type, and the positioning information is used for the display of the graphics type. Display of annotation information.

According to the above aspect and any possible implementation manner, an implementation manner is further provided, wherein the terminal-side device acquires instructions according to the guidance history input by the user, and sequentially displays them on the device display interface of the terminal-side device in chronological order. For the annotation information, the switching display of the annotation information in time sequence is realized through a pause instruction, a forward instruction or a reverse instruction input by the user.

In a fourth aspect, an embodiment of the present invention also provides a customer service system device, including a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein the processor executes The computer program is the step of realizing the video interaction method for customer service based on positioning annotation as described in the first aspect.

In the fifth aspect, an embodiment of the present invention also provides a customer service side device, including a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein the processor executes The computer program is the step of realizing the video interaction method for customer service based on positioning annotation as described in the second aspect.

In the sixth aspect, an embodiment of the present invention also provides a terminal-side device, including a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein the processor executes The computer program is the step of implementing the positioning annotation-based video interaction method for customer service as described in the third aspect.

In the embodiment of the present invention, most of the steps that need to be processed are assigned to the customer service system equipment and the customer service side equipment. The terminal side equipment only needs to transmit video information to the customer service system equipment, and receive the returned interaction information and positioning information. The display of annotation information can be realized, and the requirements for the performance of the terminal-side device itself are not high; in addition, the embodiment of the present invention completes the positioning annotation of the target entity on the basis of real-time updated image recognition, combined with the annotation of text type and image type The display of information can more accurately highlight the marked target entity, and improve the communication efficiency between terminal-side device users and customer service. In the embodiment of the present invention, not only can the equipment performance of the terminal-side equipment be reduced, but the label display effect of the terminal-side equipment can be improved, and positioning labeling can be realized more accurately, so that the interaction mode effect of video information labeling is significantly improved.

【Description of drawings】

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. Those of ordinary skill in the art can also obtain other drawings based on these drawings without paying creative labor.

Fig. 1 is a schematic diagram of the labeling method in the prior art in an embodiment of the present invention;

Fig. 2 is another schematic diagram of the labeling method in the prior art in an embodiment of the present invention;

Fig. 3 is another schematic diagram of the labeling method in the prior art in an embodiment of the present invention;

Fig. 4a is a schematic diagram illustrating the positional relationship between the label and the actual object under the condition of the first viewing angle in an embodiment of the present invention;

Fig. 4b is another schematic diagram illustrating the positional relationship between the label and the actual object under the condition of the second viewing angle in an embodiment of the present invention;

Fig. 5 is a flowchart of a video interaction method for customer service based on location annotation in an embodiment of the present invention;

Fig. 6 is a system architecture diagram of a customer service video interaction method based on positioning annotation in an embodiment of the present invention;

Fig. 7 is another flow chart of the customer service video interaction method based on positioning annotation in an embodiment of the present invention;

Fig. 8 is a schematic diagram of circled labels in an embodiment of the present invention.

【Detailed ways】

In order to better understand the technical solutions of the present invention, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

It should be clear that the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. As used in the embodiments of the present invention and the appended claims, the singular forms "a", "said" and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

It should be understood that the term "and/or" used herein is just a description of the same field of associated objects, indicating that there may be three relationships, for example, A and/or B may indicate: A exists alone, and A exists and B, there are three cases of B alone. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship.

It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present invention to describe preset ranges, etc., these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be called the second preset range, and similarly, the second preset range may also be called the first preset range.

Depending on the context, the word "if" as used herein may be interpreted as "at" or "when" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrases "if determined" or "if detected (the stated condition or event)" could be interpreted as "when determined" or "in response to the determination" or "when detected (the stated condition or event) )" or "in response to detection of (a stated condition or event)".

In the prior art, there are three interactive modes of video information labeling as follows:

Marking method one:

Circle the video screenshot: the camera on the customer service side device sends the captured video to the remote customer service instructor, and the customer service instructor marks the screenshot and sends it back to the customer service side device to overwrite the video.

A schematic diagram of its realization is shown in FIG. 1 , wherein the picture in FIG. 1 is a static picture.

Marking method two:

Continuously mark on the video: After the camera on the customer service side device sends the captured video to the remote customer service instructor, the customer service instructor will continue to click on the interaction layer in real time, and realize the video communication through the data synchronization of the interaction layer at both ends. Clients are coached in real time.

The schematic diagram of its implementation is shown in Figure 2, and the annotation is realized by marking on the video.

Marking method three:

Use Augmented Reality (AR) technology for labeling: After the camera of the customer service device sends the captured video to the remote customer service instructor, the customer service device uses the AR labeling tool to circle the first angle of view. Since the AR labeling tool Based on the spatial calculation, the circle is a virtual display in the real coordinate system, and the circle will keep corresponding to the target entity when observed from the same perspective.

The schematic diagram of its realization is shown in Fig. 3 .

There are many problems in the above three labeling methods, specifically:

The main problems of labeling method 1:

1) During the video interaction process, it is necessary to switch functional interfaces such as pictures, videos, and interactive text information, and the operation is cumbersome.

2) Relevant interactions are fragmented, and it is impossible to form a unified service process for complex interaction requirements.

3) There will be multiple types of interactive materials such as text, video, and pictures in related interactions, which puts forward more new requirements for later service management.

The main problems of labeling method 2:

1) This annotation method is a transient service method, and cannot form memorable and instructable content for complex instructive interactions.

2) This interaction method is difficult to form an association of interactive content, which poses a challenge to service traceability and service quality management.

3) Dots will affect the actual scene. When the video interactive device and the real phase move, there will be interactive visual errors, resulting in poor communication for guidance that requires complex operations.

The main problems of labeling method three:

1) This method has high requirements for the equipment on the customer service side and requires special equipment.

2) For the labeling of objects by this method, it is difficult to match the circled label with the plane of the actual object under simple tools, resulting in that the marked target cannot be seen after the viewing angle deviates. Spatial labeling requires expertise if precision is required.

Fig. 4a shows a schematic diagram circling the positional relationship between the label and the actual object under the condition of the first viewing angle of view. For comparison, FIG. 4b shows another schematic diagram illustrating the positional relationship between the label and the actual object under the second viewing angle of view. From Figure 4a and Figure 4b, it can be seen that under different viewing angles, the positional relationship between the circled mark and the plane of the actual object is still consistent (the plane where the circled mark is located and the plane near the actual object after changing the viewing angle of view is still parallel), that is, the circle mark makes adaptive adjustments with the change of the viewing angle of the video, so that the circle mark can accurately circle the actual object under different viewing angles. It is understandable that although this method of labeling works well, it has high requirements for terminal-side equipment and customer service-side equipment, and requires special equipment and professional technology for spatial labeling. In daily scenarios, the performance of common equipment and In other words, the interactive effect of using this labeling method is relatively poor.

To sum up, it can be seen that there are various problems in the labeling methods in the prior art, which lead to poor effect of the interaction mode of video information labeling.

In view of this, an embodiment of the present invention provides a video interaction method for customer service based on location annotation.

Fig. 5 shows a flow chart of the video interaction method for customer service based on location annotation in this embodiment. The customer service video interaction method based on positioning marking can be applied to a customer service video interaction system marked with positioning, and can be realized by using the customer service video interaction system marked with positioning when performing video interaction with customer service marked with positioning. The customer service video interactive system marked with positioning can be specifically applied to various computer devices, wherein the computer devices include but not limited to computers, smart phones, tablets, servers and other devices. As shown in Figure 5, the customer service video interaction method based on positioning annotation includes the following steps:

S10: The customer service system equipment receives the video information transmitted by the terminal-side equipment and forwards it to the customer service-side equipment for display, wherein the customer service system equipment includes an entity list, a recognition algorithm, and an annotation information base.

Wherein, the video information may specifically refer to streaming media information.

In an embodiment, the method for customer service video interaction based on location annotation is used in a location annotation-based customer service video interaction system, and the system includes terminal-side devices, such as mobile phones and tablets of car owners. The system also includes a customer service side. The customer service side includes customer service system equipment and customer service side systems. Among them, the customer service system equipment can be understood as cloud service platform equipment that provides data query and algorithm services, and the customer service side system can be understood as mobile phones of customer service technicians. , computer or tablet. In the customer service video interaction system based on positioning and marking, the terminal side equipment and the customer service side equipment realize the display of marking information through the customer service system equipment.

In an embodiment, the terminal device transmits the video information to the customer service system equipment, and the customer service system equipment forwards the video information to the customer service side equipment after receiving it, and displays it on the customer service side equipment.

Understandably, the customer service system equipment serves as a cloud service platform, and all service contents with high data processing and storage requirements are implemented on the customer service system equipment. Wherein, the customer service system equipment includes an entity list, a recognition algorithm, and an annotation information base. Specifically, for the special scene of the Internet of Vehicles customer service, the entities related to the vehicle are clear and limited, and the list of entities that need to be identified can be proposed in an exhaustive manner, and a dedicated identification algorithm and labeling information library can be customized in combination with the corresponding entities and environment . Combining with the Internet of Vehicles customer service scenario, it can be understood that the entity list can specifically refer to the list of entities related to the vehicle; the annotation information database stores the annotation information corresponding to the entity, so as to specifically annotate a certain entity to achieve better Labeling effect; the recognition algorithm refers to a specially designed recognition algorithm in car-related scenarios, which has a high-precision recognition function and a position recognition function. Among them, the recognition algorithm can be specifically realized through an image recognition system.

S20: The customer service device uses the entity list to select a target entity according to the input specified instruction.

It can be understood that when a user (such as a car owner) has operational difficulties with certain entities, he will complete the operation through the video guidance of the customer service technicians. Among them, it is first necessary to determine the entity to be operated by the user, that is, the target entity. Specifically, the customer service and the user can communicate the target entity in advance, and then in order to realize functions such as labeling the target entity, the customer service technicians can select the entity from the entity list provided by the customer service system equipment, and input a specified command (such as click operation ), select the target entity according to the specified instruction, and also determine the object to be labeled next.

S30: The customer service system device selects a recognition algorithm according to the target entity, and recognizes whether there is a target entity from the frame image updated by the video information according to the recognition algorithm, and if so, performs position recognition on the target entity to obtain positioning information.

Further, before step S30, that is, before the step in which the customer service system equipment selects the recognition algorithm according to the target entity, it also includes:

The customer service platform system performs image frame sampling according to the video information at intervals of a preset time interval, and obtains frame image updates of the video information, wherein the preset time interval is 1/3 second.

In one embodiment, the customer service platform system will perform image frame extraction on the video information every 1/3 second to obtain the image after each frame extraction, that is, the updated frame image of the video information. It can be understood that the customer service platform system performs image recognition and other processing on the latest updated image every time, that is, the frame image updated by the above-mentioned video information.

Wherein, the preset time interval may also be set to other time intervals. In this embodiment, a time interval of 1/3 second is adopted, which can avoid excessive recognition calculations while ensuring timeliness of recognition.

In step S30, the customer service system device will use a recognition algorithm to recognize the target entity from the frame image updated by the video information. It should be noted that the customer service system equipment will re-identify the target entity after each frame extraction according to the video information to obtain the updated frame image of the video information. For example, after every 1/3 second, the customer service system equipment will use the identification Algorithm to identify the target entity from the frame image updated by video information. Understandably, during video shooting, the viewing angle of the target entity may change. Re-identifying the target entity every 1/3 second can achieve the effect of tracking the target entity. When the observation angle of the target entity changes, the display of high-precision labeling information can be realized based on the tracking effect. For example, when an ellipse is used to circle a target entity, the ellipse that circles the target entity will change over time on the display interface of the device, and as the target entity "moves", the target will be accurately circled and marked at different viewing angles entity.

S40: Based on the annotation information base, the customer service side device obtains the interaction information edited by the customer service personnel.

In one embodiment, after the customer service technician selects the entity target involved in the communication with the user in the entity list, the interaction information edited on the customer service side device will be used on the terminal side device as The callout information is displayed. Wherein, the interaction information includes text information for interaction, such as text information and data information. Specifically, the profile information may be displayed in the form of a hyperlink, and the user may jump to an interface of detailed profile information related to the target entity after clicking the hyperlink.

Understandably, the customer service technicians can quickly complete the text information and data information to be marked on the customer service side device based on the marked information database.

Further, the device display interface of the customer service side device provides an entity operation tool for retrieving the entity and initiating a request for displaying the annotation information circled by the entity.

Among them, the entity operation tool refers to the tool for operating the entity on the device display interface of the customer service side device, such as a search tool (including a text input field for search, and the search is started by clicking the search button), and may also include editing tools for interactive information, For quick editing of interactive information. In addition, the entity operation tool may also initiate a request for displaying labeling information circled by the entity. Specifically, it can be set on the customer service side device whether to display the entity circle on the terminal device, for example, if it is selected to be yes, then when the label information is displayed on the terminal side, the target entity will be circled with a graphic such as an ellipse.

Further, the customer service side equipment is connected to the customer service system equipment, and the customer service side equipment uses the entity list and labeling information database of the customer service system equipment to implement serial number editing and add text information and data information. Understandably, the customer service side device can also number different target entities, and select appropriate text information and data information for different target entities to add.

S50: The customer service system device receives the interaction information, and synchronizes the interaction information and location information to the terminal side device and the customer service side device.

In one embodiment, the customer service system equipment will receive the interaction information sent by the customer service side equipment, and synchronize the positioning information to the terminal side equipment and the customer service side equipment together, so that labeling can be realized on both the terminal side equipment and the customer service side equipment Information display, customer service technicians can also check whether the interactive information marked by themselves is correct.

S60: The terminal-side device and the customer service-side device realize the display of annotation information based on the interaction information and positioning information, wherein the interaction information is used for displaying text-type annotation information, and the positioning information is used for displaying graphic-type annotation information.

It can be understood that the labeling information may include textual labeling, such as interaction information, or graphical labeling, such as an ellipse encircling the target entity. In an embodiment, after the terminal-side device and the customer service-side device obtain the interaction information and location information synchronously, they can display annotation information on the device display interfaces of their respective devices.

Further, the device display interface of the terminal-side device includes a video display layer and an annotation display layer, wherein the video display layer is used to display video on the device display interface according to the input video information, the annotation display layer is used to display annotation information, and the video display layer Layers and annotation display layers are consistent in display origin position and display pixels. Understandably, that is to say, the device has two display layers on the device display interface, one is used to display native video, and the other is used to display annotation information.

Further, in step S60, the terminal-side device and the customer service-side device realize the display of annotation information based on the interaction information and positioning information, including:

S61: The terminal-side device and the customer service-side device use the positioning information and the preset labeling formula to circle the target entity, so as to realize the display of labeling information circled by the entity.

In one embodiment, when the positioning information of the target entity is known, preset labeling formulas, such as ellipse generation formulas, circle generation formulas, and rectangle generation formulas, can be used to circle the target entity and generate corresponding Graphics to realize the display of labeling information circled by entities.

S62: The terminal side device and the customer service side device generate a click area on the area where the target entity is circled based on the interaction information, and the click area is used to display the interaction information and the information reply area after receiving the click instruction, wherein the information reply area is used for The terminal-side device receives the reply information input by the user.

Understandably, the user can click on the circled area of the graph, and the interactive information and information reply area will be displayed on the device display interface after clicking. Furthermore, generally only the circled annotation information is displayed on the annotation display layer, and the user can click to view the interactive information related to the target entity, and can input text in the information reply area to reply and communicate with customer service technicians. Similarly, customer service technicians can also reply to the reply message sent by the user through the message reply area.

S63: The terminal-side device and the customer service-side device implement serial number display of different target entities according to the positioning information based on the pre-edited serial numbers of different target entities.

It can be understood that the embodiment of the present invention supports marking multiple different target entities at the same time, and the user can view multiple target entities according to the sequence numbers displayed by the target entities.

Further, the positioning information specifically includes a center position (x, y) and an area pixel size (a, b).

In step S61, the terminal-side device and the customer service-side device use the positioning information and the preset labeling formula to circle the target entity to realize the display of labeling information circled by the entity, including:

其中，设定目标实体的序号显示的显示位置具体可设为(X+a/2，Y+b/2)，设定的目标实体的序号显示的显示位置也可以是其他位置，在此不作限定。可以理解地，采用以上公式能够得到圈示目标实体的椭圆形。The device on the terminal side calculates the encircled figure based on the center position (x, y) and the area pixel size (a, b) with the pixel point set as F(Xn, Yn). The formula is

Wherein, the display position of the serial number display of the set target entity can be specifically set as (X+a/2, Y+b/2), and the display position of the serial number display of the set target entity can also be other positions, which will not be set here limited. It can be understood that the ellipse that circles the target entity can be obtained by using the above formula.

Further, the customer service video interaction method based on positioning annotation also includes the following steps:

The terminal-side device acquires instructions according to the guidance history input by the user, and sequentially displays the annotation information on the device display interface of the terminal-side device in chronological order, wherein the annotation information is realized in chronological order through the pause instruction, forward instruction or reverse instruction input by the user. to switch the display.

In an embodiment, the customer service application program of the terminal-side device can also be provided with a history viewing function, which can be used to acquire instructions for guiding history input by the user (such as clicking the button for obtaining history viewing), and display annotation information in sequence on the device display interface, Generally, the earliest annotation information is displayed, and the user can switch and display the corresponding annotation information by clicking, pausing, forward, and rewinding buttons, or by inputting a specified time point.

In an embodiment, a system architecture diagram of a video interaction method for customer service based on location annotation is shown in FIG. 6 . It can be seen from Fig. 6 that the customer service system equipment receives the video information transmitted by the terminal side equipment, and obtains the positioning information through identification algorithm. The equipment on the customer service side realizes the control of the marked information according to the editing of the customer service technicians, and sends it to the customer service system equipment. Finally, the customer service system equipment synchronizes the positioning information and annotation information to the terminal side equipment and the customer service side equipment to realize the display of annotation information.

In one embodiment, as shown in FIG. 7 , a flow chart of another method for customer service video interaction based on positioning annotation is provided:

1) Relevant video customer service capabilities can be initiated by either the terminal-side device or the customer service-side device.

2) The video information acquired by the terminal-side equipment is forwarded to the customer service-side system for display through the customer service platform equipment. At the same time, the customer service platform equipment regularly draws frames from the video at an appropriate time interval, and the latest image obtained is used as the current screen image recognition application benchmark.

3) The customer service side equipment selects the target entity in the entity list according to the service requirements, and the serial number, text information and data information can be operated and edited based on the customer service side equipment to form interactive information and transmit it to the customer service system equipment.

4) After the customer service platform equipment obtains the currently marked interactive information, it first invokes the image recognition system to determine the optimal recognition algorithm according to the target entity.

5) The image recognition system uses a recognition algorithm to recognize the latest video frame image obtained in step 2), and judges whether the target entity exists, and if so, further recognizes the center coordinates (x, y) of the target entity and the area of the target entity pixel size(a,b).

6) The customer service platform device synchronizes information such as positioning information (center coordinates, area pixel size) and text information and data information received by the customer service side equipment to the terminal side and the customer service side system for labeling and display.

其中，设定目标实体的序号显示的显示位置具体可设为(X+a/2，Y+b/2)。进一步地，在标注显示层上，对圈示像素点位置的像素点进行预设透明度(如70％)的颜色显示，以突出对目标实体的标注。具体地，该圈示的过程如图8所示。7) The label display layer of the terminal side equipment and customer service side equipment adopts the optimal ellipse based on the center position (x, y) and area pixel size (a, b) of the feedback based on the positioning information and interactive information transmitted by the customer service platform equipment. Circled, the specific algorithm is as follows:

Wherein, the display position of setting the serial number display of the target entity can specifically be set as (X+a/2, Y+b/2). Further, on the label display layer, the color display of the preset transparency (such as 70%) is performed on the pixel points circled at the pixel position, so as to highlight the labeling of the target entity. Specifically, the circled process is shown in FIG. 8 .

8) When step 2 refreshes the latest image, re-execute steps 4), 5), 6), and 7).

The embodiment of the present invention adopts the labeling technology based on image recognition, which can realize the tracking and labeling of the target object. It is understandable that in common life scenarios, the terminal-side device is a general-purpose terminal device, and the ability to realize image recognition and labeling requires high-intensity computing power, and the terminal-side device is not applicable; in addition, image recognition algorithms are mainly used in specific scenarios. The recognition of specific objects in the environment is generally difficult to adapt to complex scenes, and requires the cooperation of multiple algorithms, and the computing power expenditure will increase exponentially. In the embodiment of the invention, an innovative system architecture (terminal side equipment, customer service system equipment, and customer service side equipment) is adopted, and the above problems are mainly realized through the customer service system equipment, and the equipment requirements for the terminal side equipment are not high. So far, the embodiment of the present invention adopts the tagging technology based on image recognition, which can be used in common life scenes to improve the interactive effect of video information tagging.

In the embodiment of the present invention, most of the steps that need to be processed are assigned to the customer service system equipment and the customer service side equipment. The terminal side equipment only needs to transmit video information to the customer service system equipment, and receive the returned interaction information and positioning information. The display of annotation information can be realized, and the requirements for the performance of the terminal-side device itself are not high; in addition, the embodiment of the present invention completes the positioning annotation of the target entity on the basis of real-time updated image recognition, combined with the annotation of text type and image type The display of information can more accurately highlight the marked target entity, and improve the communication efficiency between terminal-side device customers and customer service. In the embodiment of the present invention, not only can the equipment performance of the terminal side equipment be reduced, but the label display effect of the terminal side equipment can be improved, and positioning labeling can be realized more accurately, so that the interaction effect of video information labeling is significantly improved.

It should be understood that the sequence numbers of the steps in the above embodiments do not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not constitute any limitation to the implementation process of the embodiment of the present invention.

An embodiment of the present invention provides a customer service video interaction system based on location annotation, including terminal-side equipment, customer service system equipment, and customer service-side equipment, wherein the steps performed by the terminal-side equipment, customer service system equipment, and customer service-side equipment are as described in the embodiment based on The steps performed by the terminal-side device, the customer service system device, and the customer service-side device in the customer service video interaction method of positioning annotation will not be repeated here.

In the embodiment of the present invention, most of the steps that need to be processed are assigned to the customer service system equipment and the customer service side equipment. The terminal side equipment only needs to transmit video information to the customer service system equipment, and receive the returned interaction information and positioning information. The display of annotation information can be realized, and the requirements for the performance of the terminal-side device itself are not high; in addition, the embodiment of the present invention completes the positioning annotation of the target entity on the basis of real-time updated image recognition, combined with the annotation of text type and image type The display of information can more accurately highlight the marked target entity, and improve the communication efficiency between terminal-side device customers and customer service. In the embodiment of the present invention, not only can the equipment performance of the terminal-side equipment be reduced, but the label display effect of the terminal-side equipment can be improved, and positioning labeling can be realized more accurately, so that the interaction mode effect of video information labeling is significantly improved.

Those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used for illustration. In practical applications, the above-mentioned functions can be assigned to different functional units, Completion of modules means that the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be described in the foregoing embodiments Modifications to the technical solutions recorded, or equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present invention, and should be included in the scope of the present invention. within the scope of protection.

Claims (10)

1. A customer service video interaction method based on positioning annotation, characterized in that, the method comprises: The customer service system equipment receives the video information transmitted by the terminal side equipment and forwards it to the customer service side equipment for display, wherein the customer service system equipment includes an entity list, a recognition algorithm, and an annotation information library; The customer service system equipment selects the recognition algorithm according to the target entity, and identifies whether the target entity exists from the frame image updated by the video information according to the recognition algorithm, and if so, performs position recognition on the target entity, Obtaining positioning information, wherein the target entity is selected based on the entity list through a specified instruction input by the customer service side device; The customer service system equipment receives the interaction information, and synchronizes the interaction information and the positioning information to the terminal side equipment and the customer service side equipment, so as to mark information on the terminal side equipment and the customer service side equipment , wherein the interaction information is acquired through the customer service side device based on the annotation information base. 2. The method according to claim 1, characterized in that, before the customer service system equipment selects the recognition algorithm according to the target entity, further comprising: The customer service platform system performs image frame extraction according to the video information at intervals of a preset time interval to obtain updated frame image images of the video information, wherein the preset time interval is 1/3 second. 3. A customer service video interaction method based on positioning annotation, characterized in that, the method comprises: The customer service side equipment receives the video information forwarded by the customer service system equipment; The customer service side equipment uses the entity list to select the target entity according to the specified instruction input, wherein the customer service side equipment is in a communication connection with the customer service system equipment, and the customer service system equipment includes the entity list, identification algorithm and labeling information library; Based on the annotation information base, the customer service side device obtains the interaction information edited by the customer service personnel; The customer service side equipment sends the interaction information to the customer service system equipment; The customer service side equipment receives the interaction information and positioning information synchronously sent by the customer service system equipment, wherein the positioning information is that the customer service system equipment selects the identification algorithm according to the target entity, and selects the identification algorithm according to the identification The algorithm performs position recognition on the target entity identified in the frame image updated by the video information; The customer service side device realizes the display of annotation information based on the interaction information and the positioning information, wherein the interaction information is used for the display of the annotation information of the text type, and the positioning information is used for the display of the graphics type. Display of annotation information. 4. The method according to claim 3, wherein the customer service side device realizes the display of annotation information based on the interaction information and the positioning information, comprising: The customer service side device circles the target entity by using the positioning information and the preset labeling formula, so as to realize the display of labeling information circled by the entity; The customer service side device generates a click area on the area where the target entity is circled based on the interaction information, and the click area is used to display the interaction information and the information reply area after receiving a click instruction, wherein the The interaction information includes text information and data information, the data information is displayed in the form of a hyperlink, and the information reply area is used to display or reply to the reply information sent by the terminal side device; The customer service side device realizes displaying the serial numbers of different target entities according to the location information based on the pre-edited serial numbers of the different target entities. 5. The method according to claim 4, wherein the positioning information includes a center position (x, y) and an area pixel size (a, b), and the customer service side device adopts the positioning information and preset The labeling formula of the above circles the target entity to realize the display of the labeling information of the entity circle, including: According to the center position (x, y) and the area pixel size (a, b), the customer service side device calculates the encircled figure with the pixel point set as F(Xn, Yn), and the formula is

Wherein, the display position of the serial number display of the target entity is set as (x+a/2, y+b/2). 6. A customer service video interaction method based on positioning annotation, characterized in that, the method comprises: The terminal side equipment transmits the video information to the customer service system equipment, wherein the customer service system equipment includes an entity list, a recognition algorithm, and an annotation information library; The terminal side device receives the interaction information and positioning information synchronously sent by the customer service system equipment, wherein the interaction information is obtained by the customer service side device based on the annotation information database, and the positioning information is the customer service system The device selects the recognition algorithm according to the target entity, and according to the recognition algorithm, performs position recognition on the target entity identified in the frame image updated by the video information, and obtains the target entity based on the entity list through the The specified instruction input by the customer service side equipment is selected; The terminal-side device realizes the display of annotation information based on the interaction information and the positioning information, wherein the interaction information is used for the display of the annotation information of the text type, and the positioning information is used for the display of the graphics type. Display of annotation information. 7. The method according to claim 6, further comprising: The terminal-side device acquires instructions according to the guidance history input by the user, and sequentially displays the annotation information on the device display interface of the terminal-side device in chronological order, wherein the user-input pause command, forward command or reverse command realizes The switching display of the annotation information in time sequence. 8. A customer service system device, comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, characterized in that, when the processor executes the computer program, the The steps of the video interaction method for customer service based on positioning annotations described in requirement 1 or 2. 9. A customer service side device, comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, characterized in that, when the processor executes the computer program, the The steps of the video interaction method for customer service based on positioning annotations described in any one of requirements 3-5. 10. A terminal-side device, comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, characterized in that, when the processor executes the computer program, the The steps of the video interaction method for customer service based on positioning annotations described in requirement 6 or 7.

Images