CN112399125B - A remote assistance method, device and system - Google Patents

Abstract

The application discloses a remote assistance method, a device and a system, wherein the method comprises the following steps: establishing video call connection with terminal equipment based on a video call request of the terminal equipment; displaying video information from the terminal equipment, wherein the video information is obtained by carrying out video acquisition on a target object to be assisted by the terminal equipment; determining assistance information of the target object, wherein the assistance information is used for assisting a user using the terminal equipment to operate the target object; and based on an augmented reality technology, overlapping and displaying the assistance information in the video information so that the terminal equipment synchronously displays the overlapped video information. Therefore, the user can clearly describe the problems encountered by the user in a video call mode and can visually acquire the solution through the terminal device, so that the problems encountered by the user in the process of using the target object are effectively solved, and the use experience of the user is improved.

Description

A remote assistance method, device and system

technical field

The present application relates to the field of augmented reality technology, and in particular, to a remote assistance method, device and system.

Background technique

With the rapid development of science and technology, the functions of electronic products are more and more, and the operation is more and more complicated. In the process of using these electronic products, users will inevitably encounter various operational problems. Not sure how to use a certain function in electronics, etc.

Usually, when a user encounters an operation problem, the problem can be solved by calling the service phone of the product provider. However, in practical applications, users usually cannot accurately describe the problems they encounter by making calls, resulting in the inability of the assisting party to provide users with effective solutions, which in turn cannot solve the problems encountered by users, resulting in poor user experience. .

SUMMARY OF THE INVENTION

Embodiments of the present application provide a remote assistance method, device, and system, which are used to solve problems that cannot be effectively solved in the prior art when a user encounters a problem while using an electronic product, thereby affecting the user experience.

In order to solve the above-mentioned technical problems, the embodiments of the present application are implemented as follows:

In the first aspect, a remote assistance method is proposed, including:

Based on the video call request of the terminal device, establish a video call connection with the terminal device;

Displaying video information from the terminal device, where the video information is obtained by video capturing the target object to be assisted by the terminal device;

determining assistance information of the target object, where the assistance information is used to assist the user who uses the terminal device to operate the target object;

Based on the augmented reality technology, the assistance information is superimposed and displayed in the video information, so that the terminal device synchronously displays the superimposed video information.

In a second aspect, a remote assistance device is provided, comprising:

a video call module, which establishes a video call connection with the terminal device based on the video call request of the terminal device;

a display module, which displays video information from the terminal device, and the video information is obtained by the terminal device through video capture of the target object to be assisted;

a determining module, for determining assistance information of the target object, where the assistance information is used to assist a user using the terminal device to operate the target object;

The processing module, based on the augmented reality technology, superimposes and displays the assistance information in the video information, so that the terminal device synchronously displays the superimposed video information.

In a third aspect, a remote assistance system is provided, including a terminal device and a server, wherein:

sending, by the terminal device, a video call request to the server;

The server establishes a video call connection with the terminal device based on the video call request;

The terminal device and the server display video information, and the video information is obtained by the video capture of the target object to be assisted by the terminal device;

The server determines the assistance information of the target object; based on the augmented reality technology, the assistance information is superimposed and displayed in the video information;

The terminal device synchronously displays the superimposed video information.

In a fourth aspect, an electronic device is provided, the electronic device comprising:

processor; and

memory arranged to store computer-executable instructions which, when executed, cause the processor to:

Based on the video call request of the terminal device, establish a video call connection with the terminal device;

Displaying video information from the terminal device, where the video information is obtained by video capturing the target object to be assisted by the terminal device;

determining assistance information of the target object, where the assistance information is used to assist the user who uses the terminal device to operate the target object;

Based on the augmented reality technology, the assistance information is superimposed and displayed in the video information, so that the terminal device synchronously displays the superimposed video information.

In a fifth aspect, a computer-readable storage medium is provided, the computer-readable storage medium stores one or more programs, the one or more programs, when executed by an electronic device including a plurality of application programs, cause the The electronic device performs the following methods:

Based on the video call request of the terminal device, establish a video call connection with the terminal device;

Displaying video information from the terminal device, where the video information is obtained by video capturing the target object to be assisted by the terminal device;

determining assistance information of the target object, where the assistance information is used to assist the user who uses the terminal device to operate the target object;

Based on the augmented reality technology, the assistance information is superimposed and displayed in the video information, so that the terminal device synchronously displays the superimposed video information.

The above-mentioned at least one technical solution adopted in the embodiments of the present application can achieve the following beneficial effects:

According to the technical solutions provided by the embodiments of the present application, when a user encounters a problem in the process of using the target object, the user can use the terminal device to send a video call request to the server corresponding to the assisting party, and the server establishes a video call connection with the terminal device based on the video call request. , During the video call, the terminal device can capture the video of the target object, the server and the terminal device can synchronously display the video information of the target object, after that, the server can determine the assistance information for the target object, and based on the augmented reality technology The assistance information of the object is superimposed and displayed in the video information. At this time, since the terminal device and the server are in a state of a video call, the terminal device can synchronously display the superimposed video information. In this way, users can not only clearly describe the problems they encounter through video calls, but also intuitively obtain solutions through terminal devices, so as to effectively solve the problems encountered by users in the process of using the target object, and improve the user experience. user experience.

In addition, the terminal device in the embodiment of the present application does not need to have the function of augmented reality. In this way, when the user encounters a problem in the process of using the electronic product, as long as the terminal device used by the user has the functions of video capture and video call, the user can The solution can be easily obtained through the terminal device, so as to effectively solve the problems encountered and prompt the user experience.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments described in this application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

1 is a schematic flowchart of a remote assistance method according to an embodiment of the present application;

2 is a schematic flowchart of a remote assistance method according to an embodiment of the present application;

3 is a schematic structural diagram of a remote assistance electronic device according to an embodiment of the present application;

4 is a schematic structural diagram of a remote assistance device according to an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a remote assistance system according to an embodiment of the present application.

Detailed ways

Usually, when a user encounters an operation problem in the process of using an electronic product, he or she can call the service number of the product provider to solve the operation problem encountered by the user. However, because users cannot clearly describe the problems they encounter by making calls, product providers usually cannot provide users with effective solutions, resulting in poor user experience.

In the prior art, in order to improve user experience, users may be allowed to solve operational problems encountered by them through remote assistance. Generally, users can be allowed to seek remote assistance in three ways:

The first type: the user can send a video of the object to be assisted to the assisting party, and the assisting party provides assistance information to the user according to the received video.

The second type: If the object to be assisted supports remote assistance, the user can send a remote assistance request to the assisting party through the object to be assisted. Operational problems encountered.

For example, if a user encounters an operation problem when using a personal computer, he can send a remote assistance request to the assisting party through the software with the remote assistance function installed in the personal computer. After receiving and agreeing to the remote assistance request, the assisting party can directly The personal computer is operated remotely, thereby solving the operational problems encountered by the user.

The third type: the user establishes a video call connection with the assisting party through the terminal device with the augmented reality function. The assisting party provides assistance information to the user after determining the operation problem encountered by the user. The terminal device receives the assistance information from the assisting party. Afterwards, the assistance information can be superimposed and displayed on the video screen of the terminal device based on the augmented reality technology, and the user can solve the operation problems encountered by the user based on the video screen displayed in the terminal device.

However, in practical applications, the above three methods cannot effectively solve the operation problems encountered by users, specifically:

For the first method above, the facilitator usually cannot return the video containing the solution to the user. In this case, the facilitator can only inform the user of the solution through voice. However, the voice method is usually inconvenient for the user to understand. , resulting in the inability to effectively solve the problems encountered by users;

For the above-mentioned second method, since it is necessary to perform remote operation with the help of terminal devices that can support remote assistance, such as personal computers and smart phones, or software or platforms with remote assistance functions, the limitations are relatively large;

For the third method above, users need to use terminal equipment with augmented reality function to realize remote assistance, and terminal equipment with augmented reality function has not yet been popularized, so the limitation is relatively large; When a terminal device performs remote assistance, a server usually can only provide assistance information for one terminal device, and cannot provide assistance information for multiple terminal devices at the same time, so the practicability is poor.

In order to solve the above technical problems, embodiments of the present invention provide a remote assistance method, device, and system. The method includes: establishing a video call connection with the terminal device based on a video call request from the terminal device; Video information, the video information is obtained by the terminal device through video capture of the target object to be assisted; the assistance information of the target object is determined, and the assistance information is used to assist the user using the terminal device to operate the target object ; Based on the augmented reality technology, the assistance information is superimposed and displayed in the video information, so that the terminal device synchronously displays the superimposed video information.

In this way, users can not only clearly describe the problems they encounter through video calls, but also intuitively obtain solutions through terminal devices, so as to effectively solve the problems encountered by users in the process of using the target object, and improve the user experience. user experience.

In addition, the terminal device in the embodiment of the present application does not need to have the function of augmented reality. In this way, when the user encounters a problem in the process of using the electronic product, as long as the terminal device used by the user has the functions of video capture and video call, the user can The solution can be easily obtained through the terminal device, so as to effectively solve the problems encountered and prompt the user experience.

In the embodiment of the present invention, the terminal device may have the functions of video shooting and video and voice calls, but not the augmented reality function, and the target object to be assisted may be an electronic product, or other objects that can solve operation problems through remote assistance. .

In order to make those skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described The embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the scope of protection of this application.

FIG. 1 is a schematic flowchart of a remote assistance method according to an embodiment of the present application. The execution body of this embodiment may be a server corresponding to the assisting party, and the assisting party may be the provider of the target object to be assisted. The method is described below.

S102: Establish a video call connection with the terminal device based on the video call request of the terminal device.

In S102, in the process of using the electronic product, when the user encounters an operation problem, the user can use the terminal device to seek remote assistance.

When users seek remote assistance, they can use the terminal device to make a video call to the assisting party (such as the product provider of the target object to be assisted). At this time, the terminal device can send a video call request to the server corresponding to the assisting party, and the server is receiving After the video call request is received, a video call connection can be established with the terminal device based on the video call request.

In this embodiment, the server is establishing a video call connection with the terminal device, so the terminal device and the server can display the same video information synchronously.

S104: Display video information from the terminal device, where the video information is obtained by the terminal device through video collection of the target object to be assisted.

In S104, after establishing a video call with the terminal device, the server may prompt the terminal device to perform video capture of the target object to be assisted.

In an implementation manner, the server may prompt the terminal device by sending prompt information to the terminal device, wherein the prompt information may be used to prompt to align the image acquisition device of the terminal device with the target object to be assisted. After receiving the prompt information from the server, the terminal device may display the prompt information on the screen of the terminal device. The prompt information may be in a text form or a voice form, which is not specifically limited here.

After viewing the prompt information displayed on the screen of the terminal device, the user can use the terminal device to capture the video of the target object to be assisted. After the terminal device collects the video of the target object to be assisted, the video information of the target object can be obtained and displayed on the screen of the terminal device. Among them, since the terminal device and the server are in a video call at this time, the server will also synchronously display the video information collected by the terminal device.

Optionally, after the server establishes a video call connection with the terminal device, the user can also actively use the terminal device to capture the video of the target object to be assisted. After the terminal device collects video information, the terminal device and the server can display the video information synchronously.

S106: Determine assistance information of the target object, where the assistance information is used to assist a user using the terminal device to operate the target object.

In S106, after displaying the video information of the target object to be assisted, the server may determine the assistance information of the target object. In this embodiment, the assistance information may be used to assist the user to operate the target object to solve the operation problems encountered by the user, wherein the assistance information may be understood as guiding operation information, which may be a picture or a text.

When the server determines the assistance information, it can be determined based on the confirmation information of the assistance mode by the user, wherein the confirmation information may include at least one of the manual assistance mode and the intelligent assistance mode, and the confirmation information may specifically be before the server determines the assistance information. Sent by the user to the server through the terminal device.

Specifically, in an implementation manner, before determining the assistance information, the server may send text prompt information for selecting the assistance method to the terminal device, and after receiving the prompt information, the terminal device may display the prompt information on the screen, After viewing the prompt information, users can select the desired assistance method according to their actual needs. After the user selects the assistance mode, it can be regarded that the terminal device sends confirmation information for the assistance mode to the server.

For example, after receiving the prompt information from the server, the terminal device can display two buttons on the screen. The first button can be marked with intelligent assistance, and the second button can be marked with manual assistance. The user can choose to press the button according to actual needs. Pressing one of the buttons, after the user presses one of the buttons, can be regarded as the terminal device receiving the confirmation information of the assistance mode from the user, and at this time, the terminal device can send the confirmation information to the server. Wherein, the shape and style of the button and the display form of the text on the button may be various, which are not specifically limited here.

In another implementation manner, before determining the assistance information, the server may also send voice prompt information for selecting the assistance mode to the terminal device, so that the user may send the determination information to the server by voice.

For example, the server may send a voice prompt to the terminal device, and the content of the voice prompt may include: to select the intelligent assistance mode, please press "1", and to select the manual assistance mode, please press "2". After the terminal device receives and plays the voice prompt, the user can select the required assistance method by clicking the numbers on the dial pad according to their own needs, and the terminal device sends confirmation information for the assistance method to the server based on the user's choice of the assistance method.

It should be understood that, in addition to the two implementation manners described above, the user may also use the terminal device to send the confirmation information to the server in other manners, which will not be described one by one here.

It should be noted that, when the server sends the prompt information (text prompt information or voice prompt information) for selecting the assistance mode to the terminal device, it may be after establishing a video call connection and before displaying the video information of the target object, or it may be: After the video information of the target object is displayed, before the assistance information is determined, no specific limitation is made here.

After receiving the confirmation information of the assistance method from the user, when determining the assistance information according to the confirmation information, the server can judge the assistance method included in the confirmation information. If the confirmation information includes the manual assistance method, it can be determined that the user selected Manual assistance mode. At this time, the server can send the video information to the assistants through the transfer method. After viewing the video information, the assistants can manually analyze the problems encountered by the user in the process of using the target object based on the video information. Further, assistance information is determined based on the analyzed problem, and the assistance information is input into the server, so that the server can receive assistance information from the assistance personnel.

If the confirmation information includes the intelligent assistance method, the server can determine the assistance information of the target object based on the video information. The specific implementation method is as follows:

First, the video information can be identified to determine the identity of the target object to be identified and the problems encountered by the user when using the target object.

In this embodiment, when the server recognizes the video information, the recognition methods may include voice recognition and image recognition, wherein, since the user can describe the problems encountered by the user through voice, the server can Identify and determine the problems encountered by the user. Since the video information is obtained by video collection of the target object, the identification of the target object can be determined by performing image recognition on the video information.

Second, the identification of the target object and the problems encountered by the user are matched with a predetermined virtual model.

The virtual model may be predetermined by the server, and the virtual model may include the identifiers of multiple objects and the mapping relationship between multiple problems that occur when the user uses the multiple objects. The multiple objects may be multiple electronic products commonly used by the user, and the multiple problems may be problems that the user may encounter when using multiple electronic products.

After determining the identifier of the target object and the problems encountered by the user when using the target object, the server can match the identifier of the target object and the problems encountered by the user with the object identifiers and problems in the virtual model, and determine whether the virtual model includes the target The identification of the object and the problems encountered by the user when using the target object. If so, it can be explained that the problem encountered by the user can be solved by means of intelligent assistance. At this time, the corresponding assistance can be further determined based on the problems encountered by the user information.

In this embodiment, the virtual model may further include a plurality of problems that occur when using a plurality of objects and a mapping relationship between a plurality of assistance information for solving the multiple problems, wherein one question may correspond to one assistance information, Different questions can correspond to different assistance messages.

In this way, when determining the assistance information of the target object, the assistance information corresponding to the problem can be searched in the virtual model based on the problem encountered by the user, and the found assistance information can be determined as the assistance information of the target object.

Optionally, if the virtual model does not include the identification of the target object and the problems encountered by the user when using the target object, it can be explained that the method based on the intelligent assistance method cannot solve the problem encountered by the user, at this time, the video information can be It is sent to the assisting personnel by means of transfer, and the assisting personnel determines the assistance information of the target object. For the specific implementation method, please refer to the relevant content recorded above, and the description will not be repeated here.

Optionally, after the server displays the video information of the target object, when determining the assistance information of the target object, the intelligent assistance method may be used to determine the assistance information by default, and the user does not need to send confirmation information for the assistance method, that is, the server is displaying the information of the target object. After the video information is obtained, the video information can be directly identified, and then the assistance information can be determined. For the specific implementation method, please refer to the above-mentioned related content of determining the assistance information based on the intelligent assistance method, and the description will not be repeated here.

Optionally, after receiving the assistance information from the assistant, the server may also store the assistance information, the identifier of the target object and the problem encountered by the user in the virtual model, so that when other users encounter the same problem, The server may determine the assistance information in an intelligent assistance manner based on the assistance information stored in the virtual model.

S108: Based on the augmented reality technology, superimpose and display the assistance information on the video information, so that the terminal device synchronously displays the superimposed video information.

In S108, the server may use the augmented display technology to generate an augmented reality image according to the assistance information, and superimpose and display the augmented reality image in the video information, so as to generate a corresponding virtual guidance operation in the video information of the real target object. In addition, the server may also generate a corresponding voice prompt based on the assistance information, so as to simultaneously prompt the user for assistance.

It should be noted that, when the server superimposes and displays the assistance information on the video information, it may be implemented by using the existing augmented reality technology for superimposing and displaying, which will not be described in detail here.

After the server displays the superimposed video information, since the server and the terminal device are in a state of a video call, the terminal device can synchronously display the superimposed video information. In this way, the user can intuitively obtain the solution through the terminal device, thereby effectively solving the problems encountered by the user in the process of using the target object, and improving the user's use experience.

In an implementation manner, in order to facilitate the server to implement the technical solution of the present application, three modules may be provided on the server side, namely an augmented reality interaction processing module, an auxiliary information storage and input module, and an augmented reality information display module.

Taking the manual assistance method as an example, after analyzing the problems encountered by the user and determining the assistance information, the assistance personnel can input the assistance information into the assistance information storage and input module, and the assistance information storage and input module can store the assistance information. When the assistance information is superimposed and displayed in the video information, the augmented reality interaction processing module can obtain the assistance information from the assistance information storage and input module, and perform augmented reality interaction processing between the assistance information and the video information, that is, to generate an augmented reality based on the assistance information images, and overlay augmented reality images onto video information. After obtaining the superimposed video information, the augmented reality display module can display the superimposed video information in the server, and at this time, the terminal device can synchronously display the superimposed video picture. In this way, based on the interaction between the above three modules, remote assistance to the user is realized.

For example, in the process of using the washing machine, when the user encounters the problem that the washing machine cannot be started, the user can use the terminal device to make a video call to the seller of the washing machine to seek remote assistance. The server corresponding to the washing machine seller receives the user's message. After the video call, establish a video call connection with the terminal device used by the user. After the video call connection is established, the user can use the terminal device to actively capture the video of the washing machine to obtain video information. At this time, the terminal device and the server synchronously display the video information of the washing machine.

After displaying the video information of the washing machine, the server can send prompt information for selecting the assistance method to the terminal device used by the user, and the user can select the assistance method according to the prompt information. After the manual assistance mode, the server can transfer the video call to the assistant. At this time, the user can make a video voice with the assistant based on the video call to describe the operation problems encountered. It is determined that the problem encountered by the user is that the washing machine cannot be turned on. At this time, the assistant can touch the power-on button of the washing machine in the video screen, and the assistance information storage and input module in the server can be regarded as receiving the assistance information input by the assistant, and responds to the assistance. information is stored.

After that, the augmented reality interaction processing module can perform augmented reality interactive processing on the assistance information and the video information of the washing machine, that is, generate an augmented reality image according to the assistance information, and superimpose the augmented reality image to display in the video information, wherein the superimposed video information It could be: mark a blinking dotted frame around the on button of the washing machine.

After obtaining the superimposed video information, the augmented reality display module may display the superimposed video information in the server. Since the terminal device and the server are in a state of a video call, the terminal device can synchronously display the superimposed video information. In addition, the assistant can also prompt the user how to operate by voice on the basis of the video information. At this time, the user can click the power-on button of the washing machine according to the video information displayed in the terminal device and the voice prompt of the assistant, and then Solve the problem that users cannot turn on the washing machine.

To facilitate understanding of the technical solutions provided by the embodiments of the present invention, reference may be made to FIG. 2 . 2 is a schematic flowchart of a remote assistance method according to an embodiment of the present invention, which may specifically include the following steps:

S201: The terminal device sends a video call request to the server.

In the process of using the electronic product, the user can seek remote assistance through the terminal device when encountering operational problems. Specifically, the user can use the terminal device to send a video call request to the server by making a phone call.

S202: The server establishes a video call connection with the terminal device based on the video call request.

After receiving the video call request from the terminal device, the server may establish a video call connection with the terminal device based on the video call request. After the server establishes a video call connection with the terminal device, it can display the same video picture synchronously with the terminal device.

S203: The terminal device collects video of the target object to be assisted.

In S203, the server may prompt the terminal device to capture the video of the target object to be assisted, or the user may also actively use the terminal device to capture the video of the target object to be assisted, which is not specifically limited here.

After the terminal device captures the video of the target object, it can obtain the video information of the target object, wherein, in the process of video capture by the terminal device, the captured video information can be displayed on the screen of the terminal device in real time.

S204: The terminal device and the server synchronously display video information.

After the terminal device collects and displays the video information of the target object to be assisted on the screen, because the terminal device and the server device are in a video call at this time, the server can synchronously display the video information collected by the terminal device.

S205: The terminal device sends confirmation information for the assistance mode to the server.

In this embodiment, after the terminal device and the server synchronously display the video information, the server may prompt the user to select an assistance mode, wherein the assistance mode includes at least one of a manual assistance mode and an intelligent assistance mode. The user can select the assistance method according to his own needs, and the terminal device sends the user's confirmation information for the assistance method to the server. At this time, the server can receive the user's confirmation information for the assistance method.

Optionally, in other implementation manners, S205 may also be executed after S201 and before S202, or may also be executed after S202 and before S203, which is not specifically limited here.

S206: If the confirmation information includes an intelligent assistance method, determine assistance information based on the video information.

In S206, after receiving the confirmation information of the assistance method from the user, the server can determine which assistance method is included in the confirmation information. If the confirmation information includes the intelligent assistance method, the server can determine the assistance to be assisted the target object based on the video information information.

When the assistance information is determined based on the video information, it may be implemented based on a predetermined virtual model, wherein the virtual model may include the identifiers of multiple objects, multiple problems that occur when using multiple objects, and methods for solving multiple problems. Mapping relationship between multiple assistance information. For a specific implementation manner, reference may be made to the relevant content recorded in the embodiment shown in FIG. 1 , and the description will not be repeated here.

After the assistance information is determined, S208 may be performed.

S207: If the confirmation information includes the manual assistance method, receive assistance information from the assistance personnel.

In S207, if the confirmation information includes a manual assistance method, the server can send the video information to the assistants, the assistants can determine the assistance information based on the video information, and input the assistance information into the server, and the server can receive the assistance from the assistants information.

After receiving the assistance information from the assistance personnel, the server may execute S208.

S208: Based on the augmented reality technology, superimpose and display the assistance information on the video information.

After the assistance information of the target object is determined, an augmented reality image can be generated according to the assistance information, and the augmented reality image can be superimposed and displayed on the video information.

S209: The terminal device displays a video image after the assistance information and the video information are superimposed.

Since the terminal device and the server are in a video call at this time, the terminal device can display a video image in which the assistance information and the video information are superimposed.

In the technical solutions provided by the embodiments of the present application, when a user encounters a problem in the process of using the target object, the user can use the terminal device to send a video call request to the server corresponding to the assisting party, and the server establishes a video call connection with the terminal device based on the video call request. , During the video call, the terminal device can capture the video of the target object, the server and the terminal device can synchronously display the video information of the target object, after that, the server can determine the assistance information for the target object, and based on the augmented reality technology The assistance information of the object is superimposed and displayed in the video information. At this time, since the terminal device and the server are in a state of a video call, the terminal device can synchronously display the superimposed video information. In this way, users can not only clearly describe the problems they encounter through video calls, but also intuitively obtain solutions through the terminal device, so as to effectively solve the problems encountered by users in the process of using the target object and improve the user experience. user experience.

In addition, the terminal device in the embodiment of the present application does not need to have the function of augmented reality. In this way, when the user encounters a problem in the process of using the electronic product, as long as the terminal device used by the user has the functions of video capture and video call, the user can The solution can be easily obtained through the terminal device, so as to effectively solve the problems encountered and prompt the user experience.

The foregoing describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

FIG. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Referring to FIG. 3 , at the hardware level, the electronic device includes a processor, and optionally an internal bus, a network interface, and a memory. The memory may include memory, such as high-speed random-access memory (Random-Access Memory, RAM), or may also include non-volatile memory (non-volatile memory), such as at least one disk memory. Of course, the electronic equipment may also include hardware required for other services.

The processor, the network interface and the memory can be connected to each other through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or an EISA (Extended Industry Standard) bus. StandardArchitecture, extended industry standard structure) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one bidirectional arrow is used in FIG. 3, but it does not mean that there is only one bus or one type of bus.

memory for storing programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include memory and non-volatile memory and provide instructions and data to the processor.

The processor reads the corresponding computer program from the non-volatile memory into the memory and runs it, forming a remote assistance device on a logical level. The processor executes the program stored in the memory, and is specifically used to perform the following operations:

Based on the video call request of the terminal device, establish a video call connection with the terminal device;

Displaying video information from the terminal device, where the video information is obtained by video capturing the target object to be assisted by the terminal device;

determining assistance information of the target object, where the assistance information is used to assist the user who uses the terminal device to operate the target object;

Based on the augmented reality technology, the assistance information is superimposed and displayed in the video information, so that the terminal device synchronously displays the superimposed video information.

The foregoing method performed by the remote assistance apparatus disclosed in the embodiment shown in FIG. 3 of the present application may be applied to a processor, or implemented by a processor. A processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above-mentioned method can be completed by a hardware integrated logic circuit in a processor or an instruction in the form of software. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware.

The electronic device can also execute the methods shown in FIG. 1 and FIG. 2 , and implement the functions of the remote assistance apparatus in the embodiments shown in FIGS. 1 and 2 , which are not repeated in this embodiment of the present application.

Of course, in addition to the software implementation, the electronic device of the present application does not exclude other implementations, such as logic devices or the combination of software and hardware, etc., that is to say, the execution subject of the following processing procedures is not limited to each logic unit, It can also be a hardware or logic device.

An embodiment of the present application also provides a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and the one or more programs include instructions, and the instructions, when used by a portable electronic device including multiple application programs When executed, the portable electronic device can be made to execute the method of the embodiment shown in FIG. 1 and FIG. 2 , and is specifically configured to perform the following operations:

Based on the video call request of the terminal device, establish a video call connection with the terminal device;

Displaying video information from the terminal device, where the video information is obtained by video capturing the target object to be assisted by the terminal device;

determining assistance information of the target object, where the assistance information is used to assist the user who uses the terminal device to operate the target object;

Based on the augmented reality technology, the assistance information is superimposed and displayed in the video information, so that the terminal device synchronously displays the superimposed video information.

FIG. 4 is a schematic structural diagram of a remote assistance apparatus according to an embodiment of the present application. Referring to FIG. 4, in a software implementation, the remote assistance device may include: a video call module 41, a display module 42, a determination module 43 and a processing module 44, wherein:

The video call module 41, based on the video call request of the terminal device, establishes a video call connection with the terminal device;

A display module 42, which displays video information from the terminal device, and the video information is obtained by the terminal device through video capture of the target object to be assisted;

A determination module 43, which determines the assistance information of the target object, where the assistance information is used to assist the user who uses the terminal device to operate the target object;

The processing module 44, based on the augmented reality technology, superimposes and displays the assistance information in the video information, so that the terminal device synchronously displays the superimposed video information.

Optionally, the remote assistance device further includes: a receiving module 45, wherein:

The receiving module 45, before the determining module 43 determines the assistance information of the target object, receives the confirmation information of the assistance mode from the user, and the confirmation information includes at least one of the manual assistance mode and the intelligent assistance mode. kind;

Wherein, the determining module 43 determines the assistance information for the target object, including:

If the confirmation information includes a manual assistance method, receiving the assistance information from the assistance personnel, where the assistance information is determined and obtained by the assistance personnel based on the video information;

If the confirmation information includes an intelligent assistance method, then based on the video information, the assistance information of the target object is determined.

Optionally, the determining module 43, based on the video information, determines the assistance information of the target object, including:

Identifying the video information, determining the identity of the target object and the problems encountered by the user when using the target object;

judging whether the predetermined virtual model includes the identifier of the target object and the problem, and the virtual model includes the identifiers of multiple objects and the mapping relationship between multiple problems that occur when the multiple objects are used;

If so, based on the question, determine the assistance information of the target object.

Optionally, the virtual model includes a plurality of problems that occur when using a plurality of objects and a mapping relationship between a plurality of assistance information for solving the plurality of problems;

Wherein, the determining module 43, based on the problem, determines the assistance information of the target object, including:

Based on the question, searching the virtual model for assistance information corresponding to the question;

The found assistance information is determined as assistance information of the target object.

Optionally, the determining module 43 sends the video information to the assistant when the virtual model does not include the identifier of the target object and the problem;

Receive assistance information from the assistance person.

Optionally, the processing module 44, based on the augmented reality technology, superimposes and displays the assistance information in the video information, including:

generating an augmented reality image based on the assistance information;

The augmented reality image is superimposed on the video information.

The remote assistance device provided in this embodiment of the present application can also execute the methods in FIGS. 1 and 2, and implement the functions of the remote assistance device in the embodiments shown in FIGS.

FIG. 5 is a schematic structural diagram of a remote assistance system according to an embodiment of the present application. Please refer to Figure 5.

The remote assistance system shown in FIG. 5 includes a terminal device 51 and a server 52, wherein:

The terminal device 51 sends a video call request to the server 52;

The server 52 establishes a video call connection with the terminal device 51 based on the video call request;

The terminal device 51 and the server 52 display video information, and the video information is obtained by the video capture of the target object to be assisted by the terminal device 51;

The server 52 determines the assistance information of the target object; based on the augmented reality technology, superimposes and displays the assistance information in the video information;

The terminal device 51 synchronously displays the superimposed video information.

The terminal device 51 provided in the embodiment of the present invention can implement various processes implemented by the terminal device in the method embodiments of FIG. 1 to FIG. 2 , and the server 52 can implement various processes implemented by the server in the method embodiments of FIG. 1 to FIG. 2 . Repeat, and will not repeat them here.

In a word, the above descriptions are only preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

The systems, devices, modules or units described in the above embodiments may be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or A combination of any of these devices.

Computer-readable media includes both persistent and non-permanent, removable and non-removable media, and storage of information may be implemented by any method or technology. Information may be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a series of elements includes not only those elements, but also Other elements not expressly listed, or which are inherent to such a process, method, article of manufacture, or apparatus are also included. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article of manufacture, or device that includes the element.

Each embodiment in this application is described in a progressive manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, as for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the partial description of the method embodiment.

Claims (7)

1. A remote assistance method, applied to a server, includes:

establishing video call connection with terminal equipment based on a video call request of the terminal equipment;

Displaying video information from the terminal equipment, wherein the video information is obtained by carrying out video acquisition on a target object to be assisted by the terminal equipment;

determining assistance information of the target object, wherein the assistance information is used for assisting a user using the terminal equipment to operate the target object; determining assistance information of the target object based on the video information, including:

identifying the video information, and determining the identification of the target object and the problems encountered by the user when using the target object;

judging whether a predetermined virtual model comprises the identification of the target object and the problem, wherein the virtual model comprises the identifications of a plurality of objects and the mapping relation between a plurality of problems occurring when the plurality of objects are used;

if yes, determining assistance information of the target object based on the problem;

based on an augmented reality technology, the assistance information is displayed in the video information in an overlapping mode, so that the terminal equipment can display the overlapped video information synchronously;

before determining assistance information of the target object, the method further comprises:

receiving confirmation information of the user on the assistance mode, wherein the confirmation information comprises at least one of a manual assistance mode and an intelligent assistance mode;

Wherein determining assistance information for the target object comprises:

if the confirmation information comprises a manual assistance mode, receiving assistance information from an assistant, wherein the assistance information is determined by the assistant based on the video information;

if the confirmation information comprises an intelligent assistance mode, determining assistance information of the target object based on the video information;

the method further comprises the following steps:

if the virtual model does not comprise the identification of the target object and the problem, sending the video information to the assistant;

receiving assistance information from the assisting person.

2. The method of claim 1,

mapping relations between a plurality of problems occurring when the plurality of objects are used and a plurality of pieces of assistance information for solving the plurality of problems are also included in the virtual model;

wherein determining assistance information for the target object based on the question comprises:

based on the problem, searching for assistance information corresponding to the problem in the virtual model;

and determining the searched assistance information as the assistance information of the target object.

3. The method of claim 1, wherein displaying the assistance information in an overlay in the video information based on augmented reality technology comprises:

generating an augmented reality image according to the assistance information;

and displaying the augmented reality image in the video information in an overlapping manner.

4. A remote assistance system, characterized in that the system comprises a terminal device and a server, wherein:

the terminal equipment sends a video call request to the server;

the server establishes video call connection with the terminal equipment based on the video call request;

the terminal equipment and the server display video information, and the video information is obtained by carrying out video acquisition on a target object to be assisted by the terminal equipment;

the server determines assistance information of the target object; specifically, determining assistance information of the target object based on the video information includes:

identifying the video information, and determining the identification of the target object and the problems encountered by the user when using the target object;

judging whether a predetermined virtual model comprises the identification of the target object and the problem, wherein the virtual model comprises the identifications of a plurality of objects and the mapping relation between a plurality of problems occurring when the plurality of objects are used;

If yes, determining assistance information of the target object based on the problem;

based on an augmented reality technology, displaying the assistance information in an overlapping manner in the video information;

the terminal equipment synchronously displays the superposed video information;

the server is further configured to:

before determining the assistance information of the target object, receiving the confirmation information of the user on the assistance mode, wherein the confirmation information comprises at least one of a manual assistance mode and an intelligent assistance mode;

wherein determining assistance information for the target object comprises:

if the confirmation information comprises a manual assistance mode, receiving assistance information from an assistant, wherein the assistance information is determined by the assistant based on the video information;

if the confirmation information comprises an intelligent assistance mode, determining assistance information of the target object based on the video information;

the server is further configured to:

if the virtual model does not comprise the identification of the target object and the problem, sending the video information to the assistant;

receiving assistance information from the assisting person.

5. A remote assistance apparatus, comprising:

the video call module is used for establishing video call connection with the terminal equipment based on a video call request of the terminal equipment;

the display module is used for displaying video information from the terminal equipment, and the video information is obtained by carrying out video acquisition on a target object to be assisted by the terminal equipment;

the determining module is used for determining assistance information of the target object, and the assistance information is used for assisting a user using the terminal equipment to operate the target object; specifically, determining assistance information of the target object based on the video information includes:

identifying the video information, and determining the identification of the target object and the problems encountered by the user when using the target object;

judging whether a predetermined virtual model comprises the identification of the target object and the problem, wherein the virtual model comprises the identifications of a plurality of objects and the mapping relation between a plurality of problems occurring when the plurality of objects are used;

if yes, determining assistance information of the target object based on the problem; the processing module is used for displaying the assistance information in a superimposed manner in the video information based on an augmented reality technology so that the terminal equipment can synchronously display the superimposed video information;

The remote assisting apparatus further includes: a receiving module, wherein:

the receiving module is used for receiving confirmation information of the user on the assistance mode before the determining module determines the assistance information of the target object, wherein the confirmation information comprises at least one of a manual assistance mode and an intelligent assistance mode;

the determining module determines assistance information for the target object, and includes:

if the confirmation information comprises a manual assistance mode, receiving assistance information from an assistant, wherein the assistance information is determined by the assistant based on the video information;

if the confirmation information comprises an intelligent assistance mode, determining assistance information of the target object based on the video information;

the determining module is used for sending the video information to the assisting personnel when the virtual model does not comprise the identification of the target object and the problem;

receiving assistance information from the assisting person.

6. An electronic device, comprising:

a processor; and

a memory arranged to store computer executable instructions that, when executed, cause the processor to:

Establishing video call connection with terminal equipment based on a video call request of the terminal equipment;

displaying video information from the terminal equipment, wherein the video information is obtained by carrying out video acquisition on a target object to be assisted by the terminal equipment;

determining assistance information of the target object, wherein the assistance information is used for assisting a user using the terminal equipment to operate the target object; specifically, determining assistance information of the target object based on the video information includes:

identifying the video information, and determining the identification of the target object and the problems encountered by the user when using the target object;

judging whether a predetermined virtual model comprises the identification of the target object and the problem, wherein the virtual model comprises the identifications of a plurality of objects and the mapping relation between a plurality of problems occurring when the plurality of objects are used;

if yes, determining assistance information of the target object based on the problem; based on an augmented reality technology, the assistance information is displayed in the video information in an overlapping mode, so that the terminal equipment can display the overlapped video information synchronously;

Before determining the assistance information of the target object, the method further comprises:

receiving confirmation information of the user on the assistance mode, wherein the confirmation information comprises at least one of a manual assistance mode and an intelligent assistance mode;

wherein determining assistance information for the target object comprises:

if the confirmation information comprises a manual assistance mode, receiving assistance information from an assistant, wherein the assistance information is determined by the assistant based on the video information;

if the confirmation information comprises an intelligent assistance mode, determining assistance information of the target object based on the video information;

further comprising:

if the virtual model does not comprise the identification of the target object and the problem, sending the video information to the assistant;

receiving assistance information from the assisting person.

7. A computer readable storage medium storing one or more programs which, when executed by an electronic device including a plurality of application programs, cause the electronic device to perform a method of:

establishing video call connection with terminal equipment based on a video call request of the terminal equipment;

Displaying video information from the terminal equipment, wherein the video information is obtained by carrying out video acquisition on a target object to be assisted by the terminal equipment;

determining assistance information of the target object, wherein the assistance information is used for assisting a user using the terminal equipment to operate the target object; specifically, determining the assistance information of the target object based on the video information includes:

identifying the video information, and determining the identification of the target object and the problems encountered by the user when using the target object;

judging whether a predetermined virtual model comprises the identification of the target object and the problem, wherein the virtual model comprises the identifications of a plurality of objects and the mapping relation among a plurality of problems occurring when the plurality of objects are used;

if yes, determining assistance information of the target object based on the problem; based on an augmented reality technology, the assistance information is displayed in the video information in an overlapping mode, so that the terminal equipment can display the overlapped video information synchronously;

before determining assistance information of the target object, the method further comprises:

Receiving confirmation information of the user on the assistance mode, wherein the confirmation information comprises at least one of a manual assistance mode and an intelligent assistance mode;

wherein determining assistance information for the target object comprises:

if the confirmation information comprises a manual assistance mode, receiving assistance information from an assistant, wherein the assistance information is determined by the assistant based on the video information;

if the confirmation information comprises an intelligent assistance mode, determining assistance information of the target object based on the video information;

the method further comprises the following steps:

if the virtual model does not comprise the identification of the target object and the problem, sending the video information to the assistant;

receiving assistance information from the assisting person.

Images