CN104769645A - virtual companion - Google Patents

Abstract

The virtual companion described in the present invention can respond naturally to tactile input. In the case of real-time human assistance, intelligent voice communication can be carried out with the user. This invention can be applied to the care of the elderly, and promote the mental health of the elderly through continuous companionship.

Description

virtual companion

Cross References to Related Applications

Provisional Patent Application 61/774,591 filed March 8, 2013

Provisional Patent Application 61/670,154 filed July 11, 2012

The contents of the above patent applications are hereby incorporated by reference, but none of them are prior art on which this patent application is based.

Background of the invention

In China, the United States, and around the world, the number of older people is growing rapidly. Research on the health care system shows that society's attention to the elderly group is much lower than expected. At present, due to the lack of senior care professionals, the services provided by the elderly care industry cannot meet the physical and mental health needs of the elderly.

Research shows that loneliness and isolation from society and other people is a clear contributor to Alzheimer's disease, depression, reduced physical function, and even death. The severity of this problem has become increasingly apparent: In the United States, one out of every eight people over the age of 65 suffers from Alzheimer's disease (senile dementia). Elderly depression is also common: 9.4% of the elderly living alone have depressive symptoms, and the proportion of depression among the elderly living in nursing homes is as high as 42%.

In addition, studies have shown that the physical and mental health problems of the elderly also have adverse psychological and physical effects on the family members of the elderly. Due to the high cost of nursing labor in the United States, the average price is as high as 21 US dollars per hour. Many family members of the elderly cannot afford to hire round-the-clock nursing services for the elderly. They can only leave the elderly alone at home for a long time, or sacrifice their rest and working time to take care of them. elder. In the United States, the total cost of all elderly families is as high as 3 trillion US dollars per year because they hire caregivers or reduce their working hours to take care of the elderly.

On the other hand, the technological products currently on the market for the purpose of promoting social interaction for the elderly usually require users to have certain computer and Internet knowledge and operating experience. It is not suitable for the elderly, especially those who are not familiar with computers, or those who already have Alzheimer's disease.

Introduction to the invention

This invention includes: a virtual companion device; a set of multi-person collaboration and remote control of the operation process of the virtual companion; and a system for connecting the backstage staff and the virtual companion through the network.

The above functions, as well as the application advantages of each function, will be described in detail below.

picture explanation

Figure 1: 2D example of a virtual companion. The virtual companion is displayed to the user in the image of a pet, and the user can interact with the virtual companion by feeding, stroking, etc. by clicking and dragging with fingers.

Figure 2: Example of a virtual companion feeding interface: the user feeds a beverage to a virtual companion by touch.

Figure 3: 3D example of variable appearance of virtual companion: Virtual companion can change appearance according to user preference. This image shows a virtual companion in the form of a dog. In the picture, the dog's raised forelimbs are in response to the user's touch.

Figure 4: 3D example of variable appearance of virtual companion: Virtual companion can change appearance according to user preference. This image shows a virtual companion represented as a cartoon.

Figure 5: An example of the virtual partner's picture display function: the virtual partner can obtain pictures from the Internet according to the user's requirements, and display them to the user.

Figure 6: Example of the background login interface of the virtual companion system.

Figure 7: The background multi-window monitoring interface of the virtual companion system. This interface is used for remote control by the virtual companion background personnel. Each window displays the bandwidth-optimized real-time video stream sent from the client (that is, the 8 picture windows shown in the figure); the audio intensity indicator bar (corresponding to the bar area below each picture window in the figure); Reservation item list (corresponding to the list area below five of the eight windows in the figure); warning prompt information (corresponding to the area marked with X under the other three of the eight windows in the figure); current service period information (in the figure A line of text in the upper left); user and virtual partner information (a line of text above each video window); chat windows between background service personnel (the area with a scroll bar in the lower right of the figure); other employees’ current service objects Information prompt (the hourglass icon in the upper right window, the hand icon in the lower right window and the accompanying user name information); logout button (the upper right of the entire interface).

Figure 8: One-to-one service interface in the background, including: current service period information (a line of text area in the upper left of the figure); real-time video stream sent from the client (the user’s photo is used as an example in the figure); the virtual partner is currently watching the user Eye focus position (in the user's photo, between the eyes, the light-colored circular area marked with the word "Look"); historical interaction records (the double-column list on the right side of the video stream); the preset communication time between the user and the virtual partner Table (the two-column table below the historical interaction records); the image presented by the virtual partner on the client (the penguin image below the video stream); the input text box is used to input the content that needs to be read by the virtual partner (the rectangle below the penguin image area); virtual companion action control buttons (12 buttons arranged on both sides of the penguin image); virtual companion status setting area (three checkable options and three sliding setting bars on the left side of the button on the left side of the penguin image); background Service team notification bar (lower left area of the entire interface); chat window between members of the background service team (lower right area of the entire interface); miscellaneous information summary area with multi-tab structure (the label marked with 'tab' on the left side of the video stream window and the blank space below area); return to the multi-window monitoring interface button (the button marked with "Monitor All" on the upper right of the entire interface).

Figure 9: Schematic diagram of a system based on the Unified Modeling Language (UML). It shows several possible exchange methods between front-end users and back-end service personnel. The functions in the wireframes marked "Prototype 1" are already implemented, example functions used to verify the feasibility of the system.

Figure 10: Workflow diagram of background workers based on Unified Modeling Language (UML). This figure describes a series of workflows that need to be followed when a background employee logs in to the system and operates the different interfaces in Figure 6-8.

Figure 11: Schematic diagram of system deployment based on Unified Modeling Language (UML). This diagram shows an example of a possible deployment. In this example, the system connects the front-end virtual companion to the back-end control interface through a central control server. The central control server controls the virtual companion running on it by connecting multiple front-end tablet computers; it also connects multiple computers that are being used by back-office workers. In this example, in order to reduce network delay, video and audio stream information is directly transmitted between the front-end tablet computer and the background staff computer through a dedicated communication protocol (for example, RTMP protocol), without going through the central control server.

Invention in detail

Virtual companion system front end

The front end of the virtual companion is displayed to the user in the image of a pet, thereby helping the user to establish an emotional connection similar to that between people. This kind of emotional connection helps to improve the physical and mental health of the elderly living alone, and also provides a very easy-to-operate means for the elderly to obtain information from the Internet. Compared with using a desktop computer, a laptop computer or a traditional tablet computer application, when using a virtual companion, the elderly only need to issue instructions through natural language and do not need to have any computer skills. The implementation of this invention is described in detail as follows.

Virtual Companion Presentation

The virtual companion can be displayed on LCD or OLED display screens, projection or liquid crystal display panels in 2-dimensional (as shown in Figure 1, Figure 2) or 3-dimensional (as shown in Figure 3, Figure 4). The appearance of the virtual companion can be a cartoon image (as shown in Fig. 1 and Fig. 2), a lifelike animal image (as shown in Fig. 3), or between a real image and a cartoon (as shown in Fig. 4). The image of the virtual partner can be a human or human-like cartoon image, a realistic animal image, such as a penguin (as shown in Figure 1, Figure 2) or a dog (as shown in Figure 3); or any fictional image, such as a dragon, a unicorn, a ball, etc. etc.; it can even be a combination of multiple biological images (the image in Figure 4 is synthetically generated based on the images of dogs, cats, and seals). The advantage of using a fictional image is that the user will not subconsciously have a preconceived definition of the virtual partner, and will not expect that the virtual partner should/should not behave in a certain way. Therefore, when the virtual partner cannot make certain actions, the user will not be disappointed because the pre-expectation cannot be achieved; in addition, the user can also create his favorite virtual partner image according to his own preferences and imagination.

For a user, the image of the virtual companion can be unified, change randomly, or be designated by the user at the beginning of using the virtual companion according to the user's preferences, or during use. When users wish to choose a virtual companion, users will be able to choose from a range of avatar categories, including dogs, cats, aliens and more. After the user selects the type of virtual partner, he can customize the appearance color, size, and proportion of each body part of the image through an interactive interface. Another usage scenario is that the image of the virtual partner is set in advance by another user. This user may be a family member or other guardian of the elderly person for whom the virtual companion is applicable. Custom settings for virtual companions can also include behavior settings not related to appearance, which will be defined in detail below. Once the avatar of the virtual companion is set, the virtual companion can be shown to the user without any background introduction, or it can be displayed from the screen in other creative ways: for example, hatched from an egg, taken out of a gift box, or Introduce to users in other touching ways and more. Figure 4 is an example of a virtual companion avatar, which is a dog that is somewhere between a lifelike and a cartoon, and has not been customized in any way.

From a technical point of view, the display of the virtual partner can be realized through a series of 2D and 3D image rendering and video processing; Set displacement animation to realize the simulation of the virtual partner's body movements; or use a series of vector graphics to display various parts of the body, and perform motion simulation through mathematical vector transformation methods; in 3D display, the virtual partner's body consists of a series of Points, lines, and surfaces are defined, with material information attached, and material mapping is performed through a two-dimensional bitmap. In addition, the virtual companion can also be a physical robot, which is attached with a transmission device, a tactile sensor, etc. for controlling the robot's actions.

The definition of the virtual partner's appearance includes not only the definition of the static image, but also the definition of its actions. The action definition can be a series of key frame image data saved by the virtual partner in different body shapes, or a 3D vector weight definition for a 3D image. In an exemplary embodiment (as shown in FIG. 4 ), the virtual companion can be a 3D model constructed by 3D modeling software with a series of action definitions attached. First, the 3D modeling software defines the skeleton of the virtual companion image. The construction of this skeleton needs to be established with reference to the real animal skeleton, so that the running behavior of the virtual companion can be defined according to the skeleton movement during the real animal movement. In addition, additional virtual bones need to be added to the face of the virtual partner to facilitate subsequent definition of the virtual partner's facial expressions. After the skeleton definition is complete, you can define action keyframes by changing the placement of certain bones. Virtual companion actions include a default idle pose as well as other gestures such as head side to side, nod, chin lift, left foreleg lift, right foreleg lift, sad facial expression, happy facial expression, breathing, blinking, looking around Going around, wagging the tail, etc., barking, mouth movements when speaking, etc. All model appearance and motion information can be exported to the common FBX file format. When the virtual companion is displayed in the image of a real robot, the actions of the virtual companion can be defined by predefining a series of drive motor states.

During the process of using the virtual companion, the user can adjust the virtual companion's model settings (including bone size, skin texture, etc.) to show the growth process of the virtual companion from a young animal. In addition, the multi-touch interaction behavior mentioned later can also be changed in the process of simulating growth. In addition, the actions of the virtual partner defined above can also change as the virtual partner grows.

Virtual companion reacts to multi-touch

One of the key points of this invention is that the user can touch the virtual companion. When the virtual partner receives the touch signal, it will perform a series of realistic motion responses. The following exemplary embodiments assume that the virtual companion software runs on a tablet computer (iPad, Android tablet, etc.) that can accept multi-touch screen input information. When the virtual companion software runs on other platforms, it can also obtain other types of input information Do something similar. For example, in addition to the touch screen, the user can also simulate different touch states by sliding the computer mouse, clicking the left, middle, and right buttons of the mouse simultaneously or separately, clicking and dragging the mouse, and the like. When the virtual companion is displayed in the form of a physical robot, the robot can obtain input information through the attached touch input receiver. In the most basic exemplary embodiment, the virtual pet is displayed on the LCD screen of the tablet computer. This type of touchable display screen can know the area currently touched by the user by detecting the change of capacitance or resistance value in different areas. An innovative point of this invention is that when the display screen detects that multiple points on the screen are touched, the virtual partner can provide independent motion feedback for each touched part (such as the head, left forelimb, etc.), and through 3D software Merge all actions to present the user with a smooth visual effect of multiple parts moving at the same time. This split-and-merge approach to multi-touch will ultimately give users a realistic experience similar to interacting with real pets. In addition, through the analysis of information such as touch strength and track length, virtual pets can also distinguish different types of touch methods (such as stroking, poking, etc.), and thus respond differently.

The first step in realizing a virtual companion's response to touch is the monitoring of touch events. When the virtual partner runs through the game engine or other software, the query of the state of the touch sensor is implemented in the main loop of the program. In addition, the receiving of the touch signal can also be realized through a trigger mechanism, a callback mechanism, and the like.

The second step of the touch interaction is to obtain the body position of the virtual partner corresponding to the touch position. In a tablet computer or other two-dimensional touchable planar devices, the device can inform the software of the coordinate position of the touched point relative to the entire touch screen when each touch occurs. When the virtual partner is presented to the user in the form of a two-dimensional image, the virtual partner can be known by comparing the coordinates of the current touched position with the coordinate points contained in the geometric envelope representing each body part of the virtual partner which part of the body was touched. When the virtual partner is presented to the user in 3D graphics, a 3D geometric envelope needs to be established for each bone in the 3D model. For example, for the leg bones of a model, a capsule-shaped envelope (a cylinder with two hemispherical ends) can be created in the modeling software. The length, width, and height settings of the envelope ensure that the entire leg bones are completely covered inside the envelope, and there will be no relative displacement between the envelope and the bones in the model, that is, the envelope can be adjusted at any time according to the bone's Move to the corresponding position. Different body parts can define different envelope shapes. For example, for a stubby body part, the volume can be defined as a sphere. In theory, the geometry of the volume can exactly coincide with the geometry of each body part that is visible to the user as defined in the model. However, when defining the geometric shape of body parts, considering the degree of aesthetics, the geometric shape contains a large number of points, lines, and surfaces, so if the envelope is the same, the definition of points, lines, and surfaces with a large number of envelopes will be It will bring a large computational burden to the subsequent software operation. In addition, the advantage of defining the envelope as a simple geometric shape with a total volume slightly larger than the actual body part geometry is that when there is some error between the position information returned by the touch signal and the actual value, the program can still determine that the current user touches to a body part of the virtual partner. To allow for error, another approach is to define an enclosing volume that is similar in shape to the body part, but slightly larger in size, but this solution is still computationally expensive. When defining the envelope, you can define an envelope for each bone in the 3D model, or you can define the envelope only for the bones corresponding to the body parts that are set to be active in the program. In order to better receive multi-touch signals, you can also define multiple envelopes for a bone, each of which responds to a point in the multi-touch. Therefore, for a virtual companion, there is not a one-to-one relationship between touchable parts and body parts, but a many-to-one relationship. In addition, if the operating environment of the virtual companion does not support the establishment of multiple envelopes for one bone, multiple virtual bones can be created, and each bone is accompanied by an envelope to handle multi-touch. In order to improve the smoothness of the virtual companion software, it is best to pre-set all the envelopes during modeling, so that at runtime, the 3D engine that presents the virtual companion only needs to record the rotation and displacement information of each envelope frame by frame . When the virtual companion is presented in 3D, the 2D touch signal coordinates are projected onto the 3D virtual companion, and the first envelope that intersects the projection line corresponds to the body part of the virtual companion that the user wants to touch. In addition, an envelope can also be defined for other objects on the screen, and a linkage relationship between other objects and the virtual partner can be defined. When the user touches other objects, the body parts of the linked virtual partner will also respond accordingly. In addition to corresponding to the pet's body parts, the touch input can be further subdivided into "start touch" (if the touch signal did not appear in the previous frame), "stop touch" (the touch information in the previous frame, the current frame None in ), "continuous touch" (the touch information in the current frame also exists in the previous frame). In the "continuous touch" state, the virtual companion software will record the two-dimensional coordinate position of the touch signal in the previous frame, project the corresponding three-dimensional body part information, and so on.

For the cache and analysis of the touch state in a short period of time, other properties of the touch can be abstracted. In the implementation of the virtual companion software, a "touch buffer" is defined for the envelope of each body part, which is used to save the multi-frame information of the user's touch on the body part. In an exemplary embodiment, by analyzing the information stored in each touch buffer, three quantitative indicators of "persistence", "intermittent number" and "displacement" can be calculated. The persistence index starts counting from 0. When a touch signal is received in the previous frame and a touch signal is received in this frame in the touch buffer of a body part, the persistence index is increased by one. If this condition is not met, the persistence index will be decremented by 1 until it reaches 0. Therefore, when the user's touch on a body part ends, the persistence indicator will eventually drop to 0. Therefore, the persistence metric is proportional to how long the user continues to touch that body part. Intermittent numbers start counting from 0. If there is no touch signal in the current frame, but the previous frame contains a touch end signal, then the number of stutters is incremented by 1. If this condition is not met, the intermittent number is reduced by a fixed value x (x<1) after each frame until it reaches 0. For a known x, when the user taps (repeated touch, release) the screen above a certain frequency, the number of stutters will continue to increase. Therefore, the number of stutters defines the number of times the user taps a certain body part of the virtual partner in succession. In actual software writing, a fixed upper limit should be set for the number of discontinuities. The displacement can be defined as a multi-dimensional vector, which records the length of movement in each dimension, or a scalar, which records the length of a continuous touch motion track. In the two cases of vector and scalar, it can be calculated by the following method: the displacement is counted from 0, if the touch exists in the current frame and the previous frame, then in the current frame, calculate the current touched coordinate point and the previous frame The distance between touched coordinate points in one frame. When the touch ends, the displacement can be reduced by a fixed value frame by frame until it is 0, or it can be reduced by a value related to the current persistence index frame by frame until it is 0. For example, it is possible to reduce the multiplier of the persistence metric on a frame-by-frame basis. The displacement describes the length of the touch track when the user continuously touches a body part of the virtual partner. In summary, persistence, intermittent number, and displacement quantitatively describe touch events across multiple frames.

Quantitative indicators of the touch state other than persistence, intermittent number, and displacement can also be analyzed and extracted through the underlying touch input signal. The quantitative calculation methods of persistence, intermittent number, and displacement are not limited to those described above, and can also be calculated by other methods. For example: Each quantitative indicator can increase or decrease exponentially over time. Or when the indicator increases, the amount of each increase decreases with the increase of the current indicator value, so there is no need to set an upper limit for each indicator, because as the indicator value increases, the growth rate will become slower and slower. In order to simplify the calculation, multi-touch can also be simplified to single-point touch. For example, when two fingers touch the screen, the program only receives the touch information of the first finger and ignores the second finger. In addition, random noise can be added to the index value in each touch buffer, or to the increment of each index value. Adding a certain amount of random noise to the touch cache can make the virtual partner twitch or other random movements autonomously. When multiple body parts respond to touch at the same time, adding random noise can also make the transition between body parts more natural and avoid jumps.

With the help of animation mixing technology, the virtual companion converts multi-point touch information into dynamic and realistic action responses according to the quantitative index values in the touch buffer calculated in each main loop of the program. Animation mixing generally refers to a series of existing mature technologies for merging multiple independent animations of 3D animation characters into a set of continuous actions. For example, a head-down motion of the virtual partner corresponds to multiple displacement and rotation animations of the neck bones within a time period. Another movement of the virtual partner to turn the head to the right corresponds to the displacement and rotation animation of another set of neck bones. Combining the animations of these two independent actions can result in a mixed action in which the virtual partner bows his head while turning his head to the right. The range of the action corresponds to the average value of the bone displacement and rotation range in the two independent actions. Another way to achieve animation mixing is to record the relative displacement and rotation instead of defining the absolute displacement and rotation of the bones when defining a single action. The process of animation mixing is the addition of multiple relative variables, and the range of motion after mixing is larger than that of the previous implementation. In these two implementations, different weights can be set for each independent action to be mixed, so that the final mixed action is more visually biased towards the independent action with heavy weight.

An innovative point of the present invention is to use the multi-point touch signal as the input control signal for animation mixing. In an exemplary embodiment, the software predefines a series of poses for the virtual companion (idle pose in default state being one of them). The definition of each posture contains two keyframe information, the first keyframe is the state of the virtual companion in the idle posture, and the second keyframe is the state of the virtual companion in the currently defined posture. The difference between the displacement and rotation of the bones of each part of the virtual companion model between the two keyframes can be used for animation blending based on relative quantities. (If animation blending based on absolute quantities is performed, only one keyframe surface is required to define the state of each bone in the pose.) Each defined pose corresponds to the response state of the virtual partner to a continuous single-touch time. For example, a gesture of lifting the left front leg corresponds to the response state of the virtual partner when the left front leg is continuously touched. For another example, a gesture of turning the head to the right corresponds to the reaction state of the virtual partner when the left side of the face is continuously touched. Both examples above are virtual partner responses to persistent touch. Another class of gestures can be defined, corresponding to the virtual partner's response to intermittent touch. For example, a virtual partner's head tilting back corresponds to intermittent touches on the virtual partner's nose. Similarly, you can also define a series of gestures for displacement touch.

While the program is running, a weighted blend is performed on all gestures corresponding to persistent touches. The weight of each gesture is the persistence index value of the persistent touch corresponding to the gesture. That is, when a body part has not been touched for a considerable period of time, its weight is 0, and the blend action will not contain the predefined reaction gesture when this body part is touched. If the program pre-defines a series of persistent touch response gestures, and pre-defines a reasonable increase and decrease behavior of the persistent index, then the user can be presented with a realistic, touch-like entity simply by processing the persistent touch Visual effects for pets to react to when petting. For example, when the user uses one finger to continuously swipe multiple body parts of the virtual partner, the corresponding reaction gesture for each body part will be triggered. When the user leaves the body part, the weight of the reaction gesture will decrease over time. The user can thus see a series of body parts responding to touch in a coherent manner. On this basis, a series of response gestures to intermittent touch can be defined, which can be used to present a series of realistic emotional changes. For example, the posture of the virtual partner retracting the left forelimb can be defined, corresponding to the user's intermittent touch on the virtual partner's left forelimb. The weight of this gesture is proportional to the intermittent number index of the intermittent touch, so that when the user continuously and quickly clicks on the left forelimb, the intermittent number index increases, and the weight of the gesture of retracting the left forelimb in the overall mixed animation increases. Make the virtual partner show the motion of retracting the left forelimb. But when the user slowly and repeatedly clicks on the left forelimb, the intermittent number index is low, the persistence index is high, and the weight of the posture of raising the left forelimb is relatively large, so that the virtual partner presents the action of raising the left forelimb. Another way to deal with intermittent touch is to add the intermittent touch indicators received by all body parts of the virtual partner within a period of time, without separately defining the reaction gesture. The summed value corresponds to the number of random pokes on the virtual partner by the user. And define a "sad" facial expression pose. The higher the number of pokes, the higher the weight of the sad stance. In addition, other emotional gestures can be defined for intermittent touching of a specific part of the body. For example, the "happy" facial expression gesture can be defined for intermittent touches of the head. When the head is touched intermittently, the poke count will not be counted. Similarly, expression gestures can also be defined for displacement touches. For example, it is possible to define a happy expression gesture for a virtual partner of a pet dog image when the jaw is subjected to a displacement touch. Through a series of movements, the definition of expression and posture mixed with animation, the virtual companion can present the user with a realistic visual effect similar to interacting with a real pet. Users can also explore the virtual partner's response to different touches by constantly trying various possible touch locations and touch methods without being informed in advance.

In addition to controlling the action behavior of the virtual companion through multi-touch described above, the following other optional ways can also be used. For example, the weight of a gesture may be comprehensively determined by the persistence, intermittent number, and displacement of the corresponding touch. For another example, random variables can be added to the response to touch, making the response more uncertain and natural. Furthermore, touch responses corresponding to different body parts can be randomly interchanged over time. Alternatively, different touch response weights corresponding to the same body part may change over time according to a predefined random process property, or according to the current emotional information of the virtual partner. In addition, more complex touch reactions can be defined. For example, the bone position of the virtual partner can change according to the touch position of the displacement touch, so as to realize the effect that the palm position of the virtual partner changes following the touch position of the user's finger, or an effect similar to that of the virtual partner shaking hands with the user. In addition to static poses, animations with multiple keyframes can also be defined as reactions to touch. For example, a continuous side-to-side shaking motion can be defined for a virtual companion's head as a reaction to the user touching its head. When performing animation blending, in order to ensure the realistic effect of the blended motion, it is necessary to impose specific restrictions on the blending calculation. For example, when the mixed animation includes the left forelimb raising gesture of the virtual pet, it is necessary to ensure that the weight of the right forelimb raising gesture is 0 at this time, that is, ensure that the right forelimb is currently touching the ground to ensure visual rationality. In addition, the program also needs to limit the range of the mixed actions to ensure that the 3D model of the virtual pet can be displayed normally during the action. When the virtual pet is allowed on a tablet computer or other devices with accelerometer information, the current arrangement of the device can be judged through the readings of the accelerometer, and the direction of gravity can be used as one of the input control parameters for the virtual companion's action response. In addition, the camera on the device can capture the image of the user and perform gesture analysis, using gesture input as another way to interact with the virtual partner. In addition, the sound in the environment can also be received through the microphone on the device, and the sound can be used as an input to control the behavior of the virtual partner. For example, when the sound is loud, the virtual partner's ears show the motion of flapping back and forth.

In addition to the defined series of behaviors and gestures of the virtual companion described above for responding to touch, the virtual companion also pre-defines a series of spontaneous actions, such as blinking, breathing, wagging the tail, barking, jumping, talking, and so on. These actions can be performed automatically without touch events.

When the virtual companion is implemented as a physical robot, the collection of touch input information can be realized through the robot's touch sensor, and the motion mixing algorithm can be used to control the robot's motors.

Emotional control of virtual companions

In addition to the action response to the user's touch, the virtual companion can also perform action responses to the internal emotional model defined by the software. In an exemplary embodiment, the emotion model is based on the PAD model proposed by Albert Mehrabian and James A. Russel. PAD is the abbreviation of Pleasure (Pleasure)-Arousal (Activation)-Dominance (Dominance). The model uses three quantitative dimensions of pleasure, activation, and dominance to represent all emotional types. Previous research projects have applied the PAD model to define the expressions of virtual characters.

In an exemplary embodiment of the present invention, the virtual companion main program includes two sets of PAD values: long-term PAD and short-term PAD. The long-term PAD corresponds to the long-term personality traits of the virtual partner, and the short-term PAD is used to represent the current temporary emotional state. Both the initial values of the short-term PAD and the long-term PAD can be set in the following ways: 1) set to a neutral default value, 2) be selected by the user, and 3) be selected by the user's guardian. At a certain moment, the short-term PAD value can be different from the long-term PAD value, but from the overall trend, the short-term PAD value will converge to the long-term PAD value, and the convergence method can correspond to any complex functional relationship, or it can be simple and current The difference between long-term and short-term PAD is linear. Similarly, long-term PAD also tends to converge to short-term PAD, but the trend is less obvious. The convergence of long-term PAD to short-term PAD can enable the virtual partner to change personality traits with continuous acceptance and interaction with the user. Combined with the correspondence between multi-touch input and actions and emotional responses mentioned above, the short-term PAD value change of the virtual companion can be realized in the following ways:

When the number of pokes received by the virtual partner exceeds a certain threshold, the PAD's Pleasure value decreases.

When the user intermittently touches the body part of the virtual partner that defines the "happy" emoticon response. Increased pleasure.

When the user makes a displacement or continuous touch on the body part of the virtual partner that defines the "happy" expression response. Increased pleasure.

Any touch input from the user can increase the activation degree (Arousal) of the PAD and reduce the degree of dominance (Dominance). The amount of index increase or decrease caused by the touch of different body parts can be different.

Different ways of affecting PAD than described above can be defined for specific body parts of the virtual partner. For example, reduce the pleasure level when the virtual partner's eye area receives touch input, or greatly increase the pleasure level when the virtual partner's chin is touched.

The long-term PAD value can be changed not only when receiving user input, but also gradually over time. For example, the activation degree can slowly decrease over time. When there is no user touch input at night, the activation degree will not increase due to touch response. , which can ensure that the activation degree reaches the minimum in the early morning, which is consistent with the actual biological emotion change law.

When the virtual companion program includes a speech analysis function, the short-term PAD value can also vary with the intonation of the user's speech. For example, pleasure and dominance were reduced in short-term PAD when users communicated with their virtual partners in a harsh or commanding tone. In addition, the user's current breathing rate, facial emotion, etc. can be analyzed through voice and video input to obtain the user's current activation degree information, and correspondingly adjust the virtual partner's activation degree value to correspond to it.

When the short-term PAD value of the virtual partner changes, the following effects can be exerted on its behavior:

When the pleasure value is higher or lower than a certain value, the weight of the facial expression gesture corresponding to "happy" or "unhappy" in all mixed actions will be increased, so that the virtual partner can clearly understand when performing other actions. A happy or unhappy facial expression. Similarly, when the activation and dominance values change, or when the pleasure, activation and dominance change at the same time beyond a certain range, the weights of different expressions can be correspondingly increased. For example, when the degree of pleasure is lower than a certain threshold, the degree of activation is higher than a certain threshold, and the degree of dominance is at a high level, the weight of the "angry" expression can be increased.

The activation value can affect the speed of the virtual companion's breathing action. The higher the activation, the faster the breathing rate. The amplitude of the breathing action can also increase with the increase of the activation degree value. In addition, the amplitude of the virtual partner's tail wagging motion or other motions can also increase with the increase of the activation degree value.

When the number of pleasure, activation, and dominance increases, the proportion of the corresponding virtual partner's action in the action mix can also increase accordingly. For example, for a virtual companion in the form of a pet dog, the default idle pose has the tail drooping. However, when the virtual partner's pleasure, activation, and dominance numbers increase, the weight of the "tail upturned" posture will increase. After the action is mixed with the idle posture, the pet dog will show a tail upturned state.

During the operation of the virtual companion software, the age of the virtual companion will also increase, and accompanied by the change of the long-term PAD value. For example, the long-term activation value will decrease year by year. In addition, the long-term PAD value will in turn affect the growth rate of the virtual partner's age. For example, when a virtual partner has a high long-term happiness value, its age growth will slow down, or its appearance will appear younger than a virtual partner with a lower pleasantness value, such as having more vivid hair color.

Caregiving needs of virtual companions

Virtual companions can present users with virtual physical needs that grow over time. Virtual physiological needs include food needs, drinking water needs, waste excretion needs, bathing and cleaning needs, entertainment needs and so on. In addition, the spontaneous behaviors mentioned above, such as sleeping, breathing, blinking, etc., can also be defined as physiological needs. The strength of each physiological need can be defined as a numerical variable in the main program, whose value is continuously increased in the main loop. Or you can create a timer for each requirement, and its value will increase by a certain amount after a certain period of time. The increase can be different at different times of the day. The increase can also be affected by the current short-term PAD value.

The strength of a part of physiological needs can be intuitively presented to the user by changing the weight of the posture and action of the virtual partner. For example, when the need to sleep gradually increases, the weight of the virtual partner's eyelid drooping action will continue to increase. In addition, the intensity of physiological needs can also affect the short-term PAD value.

Each requirement corresponds to a variable threshold. The threshold can vary with the time of day, with the current short-term PAD value, or with some pre-defined random process variable. When the intensity of a certain demand reaches the current threshold, the virtual partner will show the corresponding behavior. For a simple demand like blinking, when the intensity exceeds the threshold, the virtual partner will blink and reset the demand intensity value to 0. Similarly, when the breathing demand exceeds its threshold, the virtual companion will perform a breathing action and set the breathing demand value to 0. When the sleep demand exceeds the threshold, the virtual partner will enter a continuous sleep state, but can be awakened by receiving external sounds and touch signals.

In an exemplary embodiment, other complex physiological needs need to be met through interaction with the user, thereby helping to establish an emotional connection between the virtual partner and the user, similar to the relationship between plants and gardeners , The relationship between pets and their owners. Research has proven that desired relationships have a positive effect on promoting mental health for users, especially older adults. When the complex requirements exceed a certain threshold, the virtual companion can prompt the user to interact by displaying specific behaviors. For example, the need for entertainment can be demonstrated by the constant jumping of a virtual companion. In addition, the virtual companion can also prompt the user by making specific sounds. For example, a food need corresponds to a virtual partner making stomach rumbling sounds, and so on.

The specific implementation examples of the exchange behavior corresponding to different nursing needs are listed below:

When the virtual partner's demand for food exceeds the threshold, the user can be prompted by the sound of stomach growling. At the same time, the virtual partner can also show an unhappy/extremely hungry expression, and can display a box with food in the program interface. container. Or, the container with food always exists in the interface. When the virtual companion has food needs, the container changes from closed to partially open or displays a flashing effect to attract the user's attention. When the user touches the container, it opens fully and displays a series of food images to choose from. Users can touch and swipe to select specific foods. As shown in Figure 1, the user can drag any food picture to the position of the virtual companion to realize the virtual feeding action. After the virtual partner receives the food, the body posture changes to the state of eating food. After the feeding state is over, the food demand value of the virtual companion decreases, and the amount of reduction can vary with the type of food selected by the user. Different food choices can also have different effects on the long-term PAD value of virtual partners. For example, when the user chooses meat, the activation of the virtual companion increases; when the user chooses vegetables, the activation decreases. Food choices can also affect the virtual growth process of a virtual partner. For example, meaty foods can give a virtual partner a more robust appearance.

When the drinking water demand of the virtual partner exceeds the threshold, the user can be prompted by making a rough breathing sound, and at the same time displaying an unhappy/thirsty expression, and at the same time a container of water appears on the program interface. When the user clicks on the container, a variety of different beverages can be displayed on the interface for the user to choose. Users can use a touch action similar to feeding to realize the water feeding action, as shown in Figure 2. Similar to feeding, water feeding behavior can also cause corresponding changes in long-term PAD values and affect the growth process of virtual partners. For example, unhealthy beverages made the virtual partner obese and decreased long-term activation, but polysaccharide beverages increased short-term PAD activation.

When the virtual partner's need for excretion exceeds the threshold, the virtual partner can enter the excretion state by itself: that is, the excrement is displayed on the screen. When excrement was present in the interface, the virtual partner's pleasure was reduced, and a series of actions were performed to show that the virtual partner did not like the bad smell. The user can touch the picture representing the excrement and slide it out of the screen; or drag a cleaning tool to the picture of the excrement to realize the cleaning behavior.

When the virtual partner's cleaning needs exceed the threshold, the virtual partner can remind the user with an itching action, or display stains on the virtual partner's body, or display the effect of 3D smoke emanating from the virtual partner's body, or on the screen Show pictures of tubs and more. Users can use touch to drag the virtual partner into the tub for cleaning. Or the cleaning function can be realized by the user touching the stains on the body of the virtual partner one by one, and the touched stains disappear one by one.

When the entertainment needs of the virtual companion exceed the threshold, the virtual companion can increase the activation index in the short-term PAD and remind the user by barking, jumping, or picking up a toy from the screen scene. Users can meet the entertainment needs of virtual companions by playing interactive games with virtual companions. The interaction mode of the game may include the multi-touch interaction described above, and the like. For example, a game could have the user take a toy out of a virtual partner's mouth through multi-touch. Interactive games can greatly increase the pleasure and activation of the virtual partner's short-term PAD, and can directly increase the pleasure and activation of the long-term PAD.

Virtual partner intelligent dialogue function and background assistance system

A key technology of the present invention is to add an intelligent dialogue function for the virtual companion. The intelligent dialogue function can be realized by artificial intelligence, and can also be remotely controlled by humans through the network system. In the case of human participation, the virtual companion acts as the avatar displayed by the remote controller on the user interface. Compared with all the use of artificial intelligence to achieve dialogue, the advantage of remotely controlling the content of the dialogue is that it can provide users with an experience closer to communicating with real people.

An artificial assistant can remotely control a virtual companion from a geographically distant location from the user. For example, the user is in the United States and the human assistant is in the Philippines or India. The artificial assistant is connected via the Internet to a tablet or other device running a virtual companion through a remote control interface running on a computer. In an exemplary embodiment, the human assistant can log into the human assistant software system through an interface as shown in FIG. 6 . After logging in, the artificial assistant can monitor the status of multiple virtual companions through the interface shown in Figure 7 . At the same time, multiple artificial companions can simultaneously monitor the same or the same group of virtual companions by logging into their respective software systems. When a virtual partner needs manual assistance, the artificial assistant software can automatically or manually enter the one-to-one control interface shown in Figure 8 by the artificial assistant. In this interface, the artificial assistant can control the behavior of the virtual companion by clicking buttons or inputting text on the interface. The series of interaction processes described above correspond to the interaction functions contained in the Prototype 1 wireframe in Figure 9. Figure 10 shows the workflow diagram of the artificial assistant. Figure 11 shows the deployment relationship of the artificial assistant software, the virtual companion software and the central control server.

When the virtual partner uses computer-generated artificial intelligence to communicate with the user, its artificial intelligence processing program can transfer the interaction tasks with high uncertainty to the remote artificial assistant. The artificial assistant software can remind the remote assistant to intervene in the current interactive process between the virtual companion and the user by displaying prompt information (similar to the prompt information display in FIG. 7 ) in the corresponding window of the multi-window monitoring interface. Alternatively, for any artificial assistant that currently does not perform one-to-one control over any virtual partner, its artificial assistant program can directly jump to the one-to-one control interface as shown in FIG. 8 . When the artificial assistant software is in the multi-window monitoring state, a series of display information can be used to prompt the artificial assistant which virtual partners are interacting with the user. These prompt information include the audio streaming content of the multi-window interface, the audio volume change received by the microphone of the tablet computer running on the virtual companion, and the like. The artificial assistant can choose to enter the one-to-one control interface of a virtual partner who is interacting through these prompts, and provide users with a better interactive experience by controlling the behavior of the virtual partner. When the artificial assistant intervenes in the middle of the communication between the virtual partner and the user, the one-to-one interaction interface will also display the conversation history between the virtual partner and the user obtained through speech recognition technology, so that the artificial assistant can timely learn the context of the current conversation. Every time the artificial assistant intervenes in an interaction process, its control information on the virtual companion can be used as the input information required for the subsequent improvement of the artificial intelligence model.

When the artificial assistant controls the virtual companion through the one-to-one control interface, the artificial assistant can know the user's current status and speech content through the video and audio stream information displayed on the interface obtained by the current virtual companion. The artificial assistant enters the language response that the virtual partner needs to make into the chat window in the interface in text form. On the virtual partner side, when it receives text input, it can read out the acquired text information in audio form through text-to-speech technology. The 3D model of the virtual partner can simultaneously make speech movements, such as the continuous opening and closing of the mouth. Actions can be played at a constant, random speed, or at a variable speed based on the content and speed of the speech. Alternatively, the lip-sync information output by the speech engine can be used to control the frequency and amplitude of speaking actions. When the virtual partner reads out the received text content, the text content can also be displayed in the form of subtitles on the screen at the same time, so that users with hearing problems can communicate with the virtual partner.

There is a delay between when the artificial assistant hears what the user is saying, and when the full response is entered via the keyboard. To reduce the time delay, the artificial assistant could be specially trained to hit the send button immediately after typing a part of the reply, and continue to enter the subsequent content while the virtual companion reads the current received content. Alternatively, the chat window of the one-to-one control interface could automatically send content to the virtual partner upon receiving the current first word (for example, after receiving a series of letters and the first space). The automatic reply content generated by the computer obtained by speech recognition and artificial intelligence can also be displayed in the dialog input window, and when the artificial assistant thinks that the response meets the requirements, it can be directly sent to the virtual partner. All interactive content input under the supervision of the artificial assistant, whether generated by the machine or manually input by the artificial assistant, can be used as the training set content for the subsequent improvement of the artificial intelligence module through machine learning. In addition, the one-to-one control interface can also display a series of optional responses for the artificial assistant, and the artificial assistant only needs to click on the most suitable response. When the artificial assistant enters by typing, the "auto-completion" function can be used to display the common words or common words beginning with the string through the letters currently typed by the artificial assistant, thereby reducing the length of the artificial assistant's input content. The artificial assistant can also realize rapid input through the information in the customer relationship management system (customer log and memo information, etc., which will be mentioned in the attachment). For example, clicking on the user name in the customer relationship management interface can display the customer name in the dialog box. Or, enter through escape characters and aliases. For example, when the artificial assistant enters "/owner", the chat input box will automatically convert /owner into the name of the corresponding customer who uses the virtual partner. Data from the CRM system can be used as input for the "autocomplete" function mentioned above.

The dialogue content between the virtual partner and the user can also be generated by an expert system or an artificial intelligence system including professional knowledge information. Expertise here includes knowledge in psychology, psychiatry, geriatric care, sociology, and more. This expert system can generate optimized reply content according to the user's current situation and speech content, for example, a question-and-answer content specially designed for patients with Alzheimer's disease to stimulate their brain activities. However, the limitation of this expert system is that it is difficult to use speech recognition technology to convert the user's arbitrary speech content into content that conforms to the input format of the expert system. For example, when a user is asked "how are you recently", the expert system can make a next judgment on three specific user responses of "I'm fine", "general" and "not so good". But when the user answers "Well, I'm not sure, it's okay", the expert system cannot make a next step of judgment. At this time, it is necessary for the artificial assistant to make a manual judgment by listening to the audio sent from the user, and select the closest standardized input (in this example, select "General"). The intervention of artificial assistants can enable users to communicate with the expert system in natural language, and reduce the possible errors of the expert system due to input mismatch. The artificial assistant can also choose to end the expert system assistance at any time during the dialogue process, or modify the output content of the expert system to make it more colloquial, or modify the parameter input required by the expert system at any time according to the current situation of the user. For example, one input parameter of the expert system is the current user's pain index. The artificial assistant can observe the user's current facial expression through the video stream, and adjust the pain index through the parameter control slider on the one-to-one interface. In addition, the PAD value similar to the emotional index of the virtual partner can also be given to the expert system, so that it can respond appropriately to the content of the user's speech under different emotions.

In an exemplary embodiment, as shown in FIG. 8, the human assistant needs to press the "Alt" key and the "Enter" key simultaneously to send out the content entered into the virtual companion. In the chat interface of multiple artificial assistants, you only need to press the "Enter" key to send. Different sending methods are adopted here to prevent the artificial assistant from mistakenly sending the content that needs to be sent to other artificial assistants to the virtual partner.

When the virtual partner uses text-to-speech technology to read out the content entered by the artificial assistant, the voice can be set to a cute child's voice and a slow speech rate, so that people with hearing difficulties can better understand the content of the virtual partner's speech. The voice, intonation, and speech speed of the virtual partner can all be changed according to the value of the emotional PAD of the virtual partner, and can also be changed according to the virtual age and appearance of the virtual partner. Or it can be set remotely by a human assistant through the assistant interface.

The artificial companion can mark the tone of voice while typing the reply in the dialog box. After the virtual partner receives the message, it can adjust its tone setting while reading the content. The virtual partner can also adjust the tone of voice according to the current emotional PAD value. For example, when the activation degree of the virtual partner is high, its speech rate and volume increase accordingly, and it often uses a rising tone at the end of the sentence.

As shown in FIG. 8 , the emotional state (PAD) value of the current virtual partner, as well as the historical interaction content with the user, the user's activity schedule, and the like are displayed in the one-to-one interface. This information allows the artificial assistant to understand the user's status, what can be mentioned in the interaction, and so on. For example, in Figure 8, according to the PAD value of the virtual partner, the artificial assistant can control the virtual partner to communicate with the user in a happy tone. During the communication, the user can ask the user about the current situation of Betty's friend Bob, or remind the user that Betty is about to have lunch, etc. .

In addition to typing the reply by the human assistant, other input methods are also possible. For example, an artificial assistant can input a reply by speaking, and the artificial assistant software can convert it into text through speech recognition function, and then send it to the virtual companion. Or, directly obtain the audio information of the artificial companion and send it to the virtual companion, and convert it into the preset language and intonation on the virtual companion through frequency conversion technology. These input processing methods make it possible that when different artificial assistants control the same virtual companion to interact with the user, the voice of the virtual companion heard by the user is always consistent and will not vary with different artificial assistants.

As shown in Figure 8, in addition to controlling the virtual partner’s voice dialogue function, the artificial assistant can also control the virtual partner’s physiological needs, emotional PAD value, facial expression, spontaneous action behavior (breathing, blinking, etc.), triggering specific actions ( Such as dancing, rolling, etc.), the artificial assistant can even record specified actions for the virtual partner by clicking and dragging different body parts in the virtual partner display frame. When the user touches the virtual companion, the virtual companion sends back the location information of the user's current touch through the network and displays it on the artificial assistant interface. The artificial assistant can also control where the virtual companion's eyes focus. In FIG. 8, the "Look" mark shows that the virtual companion's eyes are looking at the user's nose. The artificial assistant can change its position by dragging the "Look" mark, and the position information will be passed to the virtual companion program, and the program will control the eyes of the virtual companion's 3D model to look at the corresponding position.

Virtual Companion Monitoring System

A key function of the present invention is to improve the work quality and efficiency of elderly care institutions and domestic service institutions through remote monitoring. Since virtual companions can meet the needs of users for spiritual companionship and remote security monitoring, elderly care institutions only need to be responsible for dispatching a small number of personnel to complete tasks that require the presence of actual personnel, such as cleaning, cooking for the elderly, and so on.

When the artificial assistant monitors the situation of the virtual partner through the multi-window monitoring interface as shown in Figure 7, multiple artificial assistants and several supervisors can simultaneously monitor their respective interfaces and communicate with each other through the employee chat window. The eight virtual companions shown in FIG. 7 can be deployed in the same nursing home, and the artificial assistants and supervisors responsible for monitoring these virtual companions can be designated as dedicated services for the nursing home. Another feasible assignment is that the one-to-one correspondence between all virtual companions and artificial assistants is dynamic. When an artificial assistant logs into the system and starts the current working session, it can be dynamically assigned to a series of similar virtual companions. Here, similarity refers to similar user backgrounds, similar virtual partner states (for example, both in the form of pet dogs) and so on. In dynamic allocation, for each virtual companion, at least two artificial assistants can be assigned to monitor. For any two artificial assistants, the virtual companions monitored by them can partially overlap. In Figure 7, each multi-window interface can monitor 8 virtual companions. In actual implementation, the number of monitoring windows can be dynamically increased or decreased for each artificial assistant. The increase or decrease is based on the number of video streams that can be received by changing the network conditions of the artificial assistant, the size of the display used by the artificial assistant, the proficiency of the artificial assistant in monitoring multiple windows, and so on. In order to further increase the number of virtual companions that each artificial assistant can monitor, in the multi-window monitoring interface, the video window can be replaced by an abstract data window with a smaller area. A series of dynamic icons in the window show the sound received by the virtual companion, changes in video input, touch input and so on. At the same time, corresponding to each icon change, the monitoring interface can also prompt the artificial assistant through sound effects. For example, when a virtual partner receives a sudden change in sound input, the interface emits a prompt sound; when a virtual companion is touched, another prompt sound is emitted. Another basis for dynamically allocating artificial assistants is to allocate more redundant artificial assistants to virtual companions who have used artificial assistance services more frequently in the past use records, that is, there are more artificial assistants at the same time. The information of the virtual companion is displayed on the monitoring interface of the assistant. The central control system of the artificial assistant software dynamically assigns the virtual companion and human assistant currently connected to the system. The central control system can assign to each artificial assistant a part of the virtual companions that require high frequency of artificial assistance, and a part of virtual companions that require low assistance frequency, so that the workload of each artificial assistant is more evenly distributed. When the system can automatically switch any artificial assistant to the one-to-one control interface of any virtual companion as mentioned above, if the currently assigned artificial assistant is interacting with other virtual companions, the newly assigned virtual companion There will be a long waiting time. When the central control system detects that the waiting time is too long, it will automatically redistribute. The one-on-one monitoring time of the artificial assistant with each virtual companion can be recorded in a database as the basis for charging the user a time-based service fee.

The specific timing method can refer to the following information: the time when a one-to-one monitoring interface is opened or closed; the start or end time of audio and video streams; The start time of the assistant's manual recording; the comprehensive consideration of the above time points, etc.

Artificial assistants can be divided into different levels, such as paid artificial assistants, artificial assistant supervisors, volunteer artificial assistants, and even the user's family members can also become artificial assistants.

When multiple artificial assistants monitor one or a group of virtual partners at the same time, team member status indication information as shown in FIG. 7 is needed to coordinate the collaborative relationship among multiple people. Status indication can be achieved by displaying the mouse position in other artificial assistants on the monitoring interface of each artificial assistant. Different mouse shapes represent the current working status of other assistants. For example, an hourglass icon indicates that an artificial assistant is currently busy or away, and a finger icon indicates that an artificial assistant is observing the pointed location through a multi-window monitoring interface. When an artificial assistant enters the one-to-one monitoring interface, its icon will disappear from the multi-window monitoring interface of other artificial assistants, but a line of text will be displayed under the window of the virtual partner being monitored one-on-one indicating that it is being monitored by the artificial assistant (eg, "BillHelper9" in Figure 7). Before artificial assistants can use the system, they will be trained to move the mouse to a certain location on the screen when they look at it. The team member status information prompt described above can prevent multiple artificial assistants from concentrating on one virtual partner at the same time, thereby maximizing the monitoring range while ensuring redundancy. The system can also adjust the number of virtual companions monitored by each artificial assistant, thereby maximizing the number of virtual companions corresponding to an artificial assistant while ensuring a minimum response speed.

Another important function is that after a new virtual partner is connected to the system, the system can dynamically assign an artificial assistant to it, and ensure that no less than one artificial assistant is monitoring any virtual partner at any time. This dynamic allocation process consists of two phases:

1. Learning stage. When a new virtual companion is connected to the system, the system first assigns a fixed number of artificial assistants to it, and the monitoring time of each artificial assistant does not overlap. That is, only one artificial assistant monitors the virtual companion at any one time. During the learning phase, the central control server records all interactions between the virtual companion and the artificial assistant. Each record includes a timestamp, an AI ID, and an interaction score (scored automatically based on the user's satisfaction, or by the supervisor or user in charge of the AI). After a two-week learning period, the central control server analyzes all the interactive behavior data of the virtual partner, and sorts the average score of each artificial assistant that has interacted with it. Another way is to skip the learning stage, go directly to the matching stage, and dynamically rank the artificial assistants during the matching process.

2. Matching stage. When the learning phase is over, the central control server formulates an artificial assistant for the virtual companion according to the following rules: Companions assign a minimum number (e.g., 1) of human assistants that: 1) have the fewest number of virtual companions that the assistant is currently monitoring; and/or 2) within the past hour, the assistant has been and/or 3) the assistant scored the highest in the virtual companion's learning phase. When making a matching decision, the above three conditions can have different weights. When the virtual companion is in the active stage (that is, the possibility of interaction between the user and the virtual companion during this time period is high), the central control server will give priority to assigning an artificial assistant to the virtual companion that meets the following conditions: 1) the virtual companion that the assistant is currently monitoring and/or 2) the assistant had the shortest one-on-one monitoring time with all virtual partners it monitored in the past hour; and/or 3) the longest total one-on-one monitoring time with the virtual partner; 4) The assistant scored the highest in the learning phase of the virtual partner as well as in the matching phase. During the matching phase, each virtual partner will still be scored as the assistant monitors it to keep the score up to date.

The scoring of artificial assistants can also be achieved through software analysis of the user's voice intonation. The tone of voice that contains higher activation and pleasure indicates that the user is more satisfied with the current interaction. In addition, other sensor information can also be used to score, such as user skin conductivity, skin color, changes in pupil size, and so on. Or it can analyze the user's touch on the screen to quantify indicators and the like.

Since the virtual partner can obtain the video and audio input of the user terminal through the camera and microphone of the running device, in order to reduce the user’s doubts about privacy, when the virtual partner enables high-resolution input, it is necessary to give the user a certain prompt to indicate that it can be remotely monitored. People hear and see. This prompt information can be transmitted to the user in a natural way through the posture change of the virtual partner. For example, the collar of the virtual partner's neck lights up, the color of the eyes of the virtual partner changes, or the virtual partner presents a state of wide-opening eyes, etc. In an exemplary embodiment, the virtual companion's sleep and wake state surface current video, audio stream feeds are turned off or on. This display method allows users to intuitively understand whether there is audio and video information collection at present without technical background. Other information that does not involve privacy, such as ambient volume changes, light changes, screen touch input, etc., can be collected and uploaded to the background at any time, regardless of whether the current video or audio input is enabled.

Another important service function of the remote artificial assistant is to be able to contact third-party personnel in time when the user has an emergency or needs manual assistance. The third party may be a staff member of a nursing home where the virtual companion is deployed, or a family member of the user, and so on. Contact details of third party personnel are kept in the system database along with log records related to other virtual companions. In FIG. 8, the contact information may be displayed in the area marked with "tab". When the artificial assistant clicks on the contact information, the system can automatically dial the corresponding phone number, or pop up an email window.

The remote artificial assistant system can also be used to provide remote technical support to the virtual companion. One of the purposes of technical support is to ensure that the hardware and software of the virtual companion can continue to operate normally. The virtual partner periodically sends status information to the server through the network. An artificial assistant monitoring the virtual companion can receive status information in real time. The artificial assistant can also issue specific instructions to the virtual partner to obtain more information, such as a screenshot of the current device the virtual partner is running on, and so on. The artificial assistant can also send instructions for the virtual companion software to execute, such as "restart the virtual companion software", "change the volume of the speaker", "restart the running device" and so on. When the virtual companion software is not running normally, or during the daily routine operation and maintenance phase, the virtual companion can be restored to its initial operating state by executing remote instructions. In terms of program design, a simple but stable daemon program can be used to monitor and control the operation of the main program. The main program is responsible for more complex and more fault-prone functions such as the visual presentation and interactive control of the virtual companion. By receiving remote commands, the daemon can shut down, restart, or perform other operational problem analysis operations on the main program. The daemon can be periodically called by the main program or the operating system to send and receive status information and instructions to be executed.

Other functions of the virtual companion

In addition to the functions mentioned above, the virtual companion can also implement other functions to enrich the user experience and improve the quality of life of the user.

The virtual companion can broadcast news, weather or any other text-based information on the Internet to the user on demand. Users can express their needs to the virtual partner through voice, such as "news about the general election", "the weather in Tokyo", etc., and the virtual partner can understand the user's needs through voice recognition or with the help of artificial assistants. After the virtual companion understands the needs of the user, it can take out newspapers or documents, and at the same time the virtual companion software searches for the corresponding content on the Internet through the background network connection. After capturing the content, the virtual companion reads the content aloud using text-to-speech technology. In addition to the above-mentioned function of obtaining and reading information according to the user's requirements, the user's family can also provide the virtual companion with content that the user may be interested in in advance, and the virtual companion can periodically read the content spontaneously to the user based on these contents. For example, the virtual companion can remind the user that "your son posted a new status online today" and then read the status content.

The virtual companion can also provide image information to the user in a similar manner. The way of displaying the image information may be that the virtual companion displays a photo frame or photo album to the user, and the photo frame presents the pictures required by the user (as shown in FIG. 5 ). In addition, the virtual partner can also download photos uploaded by family members from the download address preset by the user's family members and display them to the user. Similarly, the virtual companion can also display audio (playing through a virtual radio or gramophone) and video materials for the user. Materials can be downloaded from sites such as YouTube according to the user's needs, or uploaded to designated locations in advance by the user's family members. In addition, users can also issue instructions to their virtual partners to video chat with family members through voice. The virtual companion will display a photo frame or TV on the screen after receiving instructions, and display family video images inside it. The family member’s video chat account information is pre-stored in the virtual partner’s memo information in advance.

The virtual partner can detect the user's breathing rate through a camera or microphone, and synchronize the virtual partner's breathing rate with it. The virtual companion can then help stabilize the user's mood by gradually reducing her own breathing rate. This feature can be used to calm irritable Alzheimer's patients or agitated children with autism.

Actual objects can be used to interact with virtual companions through technologies similar to virtual reality. For example, we found in the process of developing products that users like to share common experiences with their virtual partners. For example, users hope to take a nap with their virtual partners. Virtual companions can provide shared experience of taking medication and other aspects through virtual reality technology. In an exemplary embodiment, the user may place a container of medication that they need to take in front of the camera. The virtual companion can use image recognition technology or a remote artificial assistant to help identify the current object as a medicine container, and after checking with the user's preset schedule information to confirm that the user needs to take the medicine at this time, it will appear on the screen display of the virtual companion. A piece of virtual food. The virtual food can be in the shape of the medicine taken by the user, or in other shapes, for example, a piece of bone. When the user starts to take the medicine, the virtual partner detects this behavior through similar technology and starts to show the action of eating food and the happy expression. By connecting the virtual food with the real medicine, the user realizes the function of feeding the virtual pet by taking the medicine regularly. This enables users to develop a habit of taking medicine at regular intervals while strengthening their sense of personal responsibility, and the virtual partner provides positive feedback for users by showing happy expressions of getting food, and promotes users' taking medicine experience.

In addition, the user can interact with the virtual companion by showing a card with a specific pattern to the camera of the virtual companion's device. After the virtual companion program detects the content on the card, it will display the corresponding virtual item in the virtual environment on the screen. By moving the position of the card in the actual environment, the user can control the position movement of the corresponding item in the virtual environment, thus providing a new way for the user to interact with the virtual partner.

Some tablet computers have near-field communication or RFID functions. Users can interact with virtual companions by placing communication tags near the tablet computers, and the virtual companion software receives the tag information and displays the corresponding items on the screen. The tablet on which the virtual companion resides may include a tag collection card slot for receiving approach communication tags. The collection card slot is located near the entry communication module of the tablet computer, and it is necessary to ensure that when the user throws a tag into the card slot, the tag can be read by the entry communication module. When the virtual companion software detects the incoming communication reading event, it will display on the screen that the corresponding items are thrown into the environment. When the tablet computer does not have the near-field communication function, the similar effect of throwing objects into the virtual environment where the virtual partner is located can also be realized by detecting the event of the card with the icon slipping through the camera.

The virtual companion can also be implemented in the form of a webpage, and the artificial assistant can only perform one-to-one control on the webpage version of the virtual companion within a limited period of time. This realization is mainly convenient for users who first contact the virtual companion without hardware facilities. Try it out. For users who have not registered a virtual companion account, they can open the virtual companion trial webpage in a browser and click the start button. At this time, the user will be in a waiting state. If there is only one web version user waiting, the system will assign an idle artificial assistant to it and start the trial. If there are currently multiple web version users waiting, the system will put the users into the waiting queue according to the time when the user clicks the start button event. Start the trial period for the second user in the queue. During the trial process, the virtual companion receives video and audio information through the camera and microphone of the computer currently used by the user. If the computer does not support touch input, a mouse click can be used to simulate the behavior of the user touching the virtual companion. After the user's trial period begins, a countdown timer can be triggered to start counting down from a fixed time (for example, 2 minutes). After the countdown ends, the trial will end automatically. The human assistant can also end the current trial through the assistant interface before the countdown ends. After the current trial ends, if there are still waiting users in the queue, the system will automatically connect the next user to the artificial assistant service.

The virtual companion system also includes automated training capabilities for human assistants. When the human assistant is being trained automatically, a special version of the human assistant software is used. This version of the software includes simulated virtual companion video, audio input, and simulated user touch input to the virtual companion. All operations of the trained artificial assistant, including keyboard input, mouse movement, click, etc., will be recorded by the software and used for subsequent training evaluation.

Improvements to existing inventions

Nursing homes, nursing communities and other institutions have long been in a state of shortage of nursing staff, and the staff turnover rate is high. In some nursing homes in the United States, the turnover rate of nursing staff is close to 100%. Therefore, the elderly in nursing homes spend most of their time in a state of being unaccompanied. The resulting isolation from society has a negative impact on the emotional and mental state of the elderly, and can accelerate the development of diseases of the elderly such as Alzheimer's disease. Because care workers are expensive and have limited time; using pets to accompany the elderly requires additional labor to take care of pets, so a series of solutions that use artificial intelligence to accompany the elderly have emerged. Several possible solutions are listed below, and their differences from the present invention are discussed.

Paro (http://www.parorobots.com) is a physical robot designed to solve the loneliness of the elderly. Due to its high manufacturing cost and high price, it is difficult to popularize on a large scale. In addition, its movements are limited to simple crawling, moving limbs, etc., and its facial expressions are also limited, and it does not have the language ability of real human intelligence.

A virtual pet toy for children (eg, US Patent Application No. 2011/0086702) is an electronic game developed for children. Due to its complicated functions and cumbersome operation, it is not suitable for the elderly to use. In this category of inventions, some can receive user's touch, mouse click (for example, US Patent Application No. 2009/0204909; Talking Tom: http://outfit7.com/apps/talking-tom-cat). However, its responses to input are mostly pre-generated limited types, or simple imitation of user input, which makes users feel bored after seeing repeated content after using it for a period of time. Compared with these inventions, the responses of the present invention to touch and mouse click input are dynamically generated and visually more natural.

Currently existing virtual assistant systems that can handle voice input can only simply repeat the user's input as output (such as Talking Tom) or recognize the user's voice input through voice recognition technology and generate output content through artificial intelligence technology (such as the US Patent No. 6,772,989; US Patent Application No. 2006/0074831; Siri: US Patent Application No. 2012/0016678). Since speech recognition and artificial intelligence technologies are not yet mature, it is difficult for the above-mentioned patent to achieve the effect of dialogue with a real person through the intervention of an artificial assistant in this patent.

Human-assisted intelligent systems (such as US Patent Application No. 2011/0191681) are used in retail customer service systems, remote monitoring, etc. But it has not been applied to scenarios like virtual companions.

Other applications of the invention

When a user uses a virtual companion, usage statistics may be collected and used as a source of analytical data for diagnosing the user's physical and mental health (eg, functional decline, Alzheimer's development). For example, when older people have symptoms of depression or social resistance, they interact less frequently with their virtual partners. Therefore, virtual companion usage statistics can serve as a more accurate, non-invasive source of clinical diagnostic data.

The invention can also be applied to other populations, such as young people. Due to the intervention of the artificial assistant, the present invention has a strong entertainment function. In addition, it can also be an assistant for the user's life, manage time arrangements or complete other network-related tasks.

The invention can also be applied to the nursing and treatment of children with autism. Autistic children tend to communicate with non-human subjects. Since virtual companions can be displayed in the image of animals, and the existence of artificial assistants gives them a way of communication similar to humans, children with autism are easy to accept virtual companions, and gradually adapt to communicating with humans with the help of virtual companions.

The invention can also be used as a toy for children. In this case, more game interaction functions need to be added, and richer 3D model poses should be added to correspond to the states of the user when taking care of the virtual partner in different ways.

The invention can be used by orthodontists to provide daily dental care training and reminders for their patients. In an orthodontic cycle, multiple virtual companions can be pre-set to be responsible for different stages of the cycle, and each virtual companion is pre-set in advance what needs to be informed to the user by voice or other means in this stage, so as to carry out the whole process fully automatically. Cycle with promises and reminders.

The present invention's response to multi-touch can be applied to various 3D models, not limited to 3D human or animal models. For example, a multi-touch reaction can be defined for a 3D flower model.

The invention can be used for the control of physical robots. For example, a tablet computer running a virtual companion can be connected to a mechanical part, and the movement of the mechanical part can be controlled by commands issued by the virtual companion.

Additional physical components can be added to the tablet running the virtual companion to enhance the user's visual experience. For example, when the virtual companion is shown as a pet dog, a kennel-shaped outer frame can be added around the tablet.

In addition, physical parts with special haptic effects can be added to the tablet running the virtual companion to enhance the user's tactile experience. For example, a soft, plush shell can be added to the tablet, and the user can insert a finger into the opening of the shell.

The virtual companion can also tell whether a touch input is due to the presence of liquid on the screen by analyzing the nature of the touch input. Because liquid-triggered touch input usually has rapid fluctuations in intensity and position. After the virtual partner detects the liquid touch, the interaction of the virtual partner with the liquid (such as rain) can be displayed in the virtual scene.

Claims (20)

1. A virtual companion device, comprising: Ways to demonstrate a virtual companion; A method to detect user input; A method for altering the presentation of a virtual companion based on detected user input. 2. The device as claimed in claim 1, wherein: The detected user input is touch; Changing the display method of the virtual partner needs to read the position input by the user and determine which body part of the virtual partner it corresponds to; The method of changing the presentation of the virtual companion includes moving the body parts of the virtual companion. 3. The device of claim 1, wherein: A virtual companion is an animated figure that can exhibit one or more actions, each representing the virtual companion's response to a stimulus; When multiple stimuli are received, multiple corresponding actions may be blended according to weights. 4. The device of claim 3, wherein: A virtual companion can display a virtual item with specific content; the content is retrieved from a remote database. 5. The device of claim 2, further comprising: A method by which a virtual companion can be controlled remotely; A method that enables a virtual companion to perform voice output under remote control. 6. The device as claimed in claim 5, wherein: Virtual companions can take on non-human forms. 7. The device of claim 1, further comprising: A method of stimulating the physical needs of a virtual companion; A method for representing the biological needs of a virtual companion through hybrid animation. 8. The device of claim 1, further comprising: A method of identifying physical items through image recognition and converting them into virtual items that can be exchanged by virtual companions. 9. The apparatus of claim 8, further comprising: A solid physical device is used to guide the user to place the physical item in a specific location for identification; this physical structure can also be used to support a fixed virtual companion display device. 10. The device of claim 1, further comprising: A physical physical device is attached around or near the virtual companion display as part of the virtual companion display's visuals. 11. The device of claim 1, wherein: The method of detecting user touch input adopts capacitive touch screen; The virtual companion can respond to touch signals triggered by the presence of liquid on the screen. 12. A method of controlling one or more virtual companions, comprising: The virtual companion transmits data to the server; The server forwards the received data sent by the virtual companion to the networked computer; Display the status of each virtual companion on a networked computer; Users of networked computers can select any virtual companion and open a detailed view including its detailed status; Sending video and audio streaming data from virtual companions selected to display detailed views to networked computer users; Internet-connected computer users can send commands to selected virtual companions. 13. The device of claim 12, wherein: The virtual companion in the non-detailed view state sends less accurate information to the networked computer than the virtual companion in the detailed view; When a virtual partner is selected to display a detailed view on the networked computer, the virtual partner's 3D model display side is more focused. 14. The device of claim 12, wherein: Each controlled virtual companion is a device described in claim 5 . 15. The device of claim 12, wherein: The commands sent to the virtual companion are generated by artificial intelligence; Commands generated by artificial intelligence can be modified and approved by users using internet-connected computers before being sent. 16. A system in which multiple persons control multiple avatars, comprising: Multiple avatars, each avatar has its own records of historical events and personality traits; Multiple people, each person can remotely control each avatar in a certain way through a networked computer; A means by which an event history can be entered by a human avatar; A way for one to read out the event history of each avatar. 17. The system of claim 16, wherein: Each avatar is a device described in claim 5. 18. The system of claim 16, wherein: Each person controls the avatar in the manner described in claim 12. 19. The system of claim 16, further comprising: A way to dynamically allocate the corresponding relationship between the controller and the controlled avatar in order to maximize the labor efficiency. 20. The system of claim 19, further comprising: A way to record everyone's performance while controlling each avatar; A method of assigning controllers and avatars based on historical performance during dynamic assignments in order to maximize overall performance.