CN115937373B - Avatar driving method, apparatus, device and storage medium - Google Patents

Abstract

The present disclosure provides a virtual image driving method, device, equipment and storage medium, and relates to the technical field of artificial intelligence, especially to the technical fields of virtual human, metaverse, augmented reality, virtual reality, mixed reality, extended reality and other technical fields. The specific implementation plan is: receiving an input data stream. The input data stream includes multiple timing frames. Any timing frame is associated with a facial posture. The facial posture includes a first part posture and a second part posture. For any of the multiple timing frames, In a target timing frame, the first posture transformation coefficient is determined based on the initial reference posture and the first part posture; the initial reference posture is updated according to the first posture transformation coefficient to obtain the updated reference posture; and the third posture is determined based on the updated reference posture and the second part posture. Two posture transformation coefficients; driving the virtual image according to the second posture transformation coefficient.

Description

Virtual image driving method, device, equipment and storage medium

Technical field

The present disclosure relates to the technical field of artificial intelligence, especially to the technical fields of virtual human, metaverse, augmented reality, virtual reality, mixed reality, extended reality, etc., and specifically to a virtual image driving method, device, equipment and storage medium.

Background technique

With the development of computer technology and Internet technology, various functional services in life, entertainment and other aspects can be provided through virtual images. For example, some avatars provide visual display services. How to make the facial expressions made by avatars more realistic is an urgent problem that needs to be solved.

Contents of the invention

The present disclosure provides a virtual image driving method, device, equipment and storage medium.

According to an aspect of the present disclosure, an avatar driving method is provided, including: receiving an input data stream, wherein the input data stream includes a plurality of time series frames, any one of the time series frames is associated with a facial gesture, and the facial gesture includes a first part The posture and the second part posture, the second part posture includes multiple second sub-part postures, the correlation between any multiple first part postures is smaller than the correlation with the multiple second sub-part postures, for multiple For any target timing frame in the timing frame, determine the first posture transformation coefficient according to the initial reference posture and the first part posture; update the initial reference posture according to the first posture transformation coefficient to obtain the updated reference posture; based on the updated reference posture and the second posture Part posture, determine the second posture transformation coefficient; and drive the virtual image according to the second posture transformation coefficient.

According to another aspect of the present disclosure, an avatar driving device is provided, including: an input data stream receiving module, configured to receive an input data stream, wherein the input data stream includes a plurality of time series frames, any one of the time series frames and facial gestures Correlation, the facial posture includes a first part posture and a second part posture, the second part posture includes multiple second sub-part postures, and the correlation between any multiple first part postures is smaller than the multiple second sub-part postures. The correlation between , used to update the initial reference posture according to the first posture transformation coefficient to obtain the updated reference posture; the second posture transformation coefficient determination module is used to determine the second posture transformation coefficient according to the updated reference posture and the second part posture; the avatar driving module , used to drive the virtual image according to the second posture transformation coefficient.

According to another aspect of the present disclosure, an electronic device is provided, including: at least one processor and a memory communicatively connected with the at least one processor. The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor, so that at least one processor can execute the method of the embodiment of the present disclosure.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform methods of embodiments of the present disclosure.

It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of the drawings

The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. in:

Figure 1 schematically shows a system architecture diagram of an avatar driving method and device according to an embodiment of the present disclosure;

Figure 2A schematically shows a flow chart of an avatar driving method according to an embodiment of the present disclosure;

Figure 2B schematically shows a schematic diagram of the posture of the chin-lip region;

Figure 3 schematically shows a schematic diagram of an avatar driving method according to another embodiment of the present disclosure;

Figure 4 schematically shows a schematic diagram of determining the third posture transformation coefficient of the avatar driving method according to yet another embodiment of the present disclosure;

Figure 5 schematically shows a block diagram of an avatar driving device according to yet another embodiment of the present disclosure; and

FIG. 6 schematically shows a block diagram of an electronic device that can implement the avatar driving method of an embodiment of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the present disclosure are included to facilitate understanding and should be considered to be exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. The terms "comprising," "comprising," and the like, as used herein, indicate the presence of stated features, steps, operations, and/or components but do not exclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used here should be interpreted to have meanings consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

Where an expression similar to "at least one of A, B, C, etc." is used, it should generally be interpreted in accordance with the meaning that a person skilled in the art generally understands the expression to mean (e.g., "having A, B and C "A system with at least one of" shall include, but is not limited to, systems with A alone, B alone, C alone, A and B, A and C, B and C, and/or systems with A, B, C, etc. ).

Figure 1 schematically shows the system architecture of an avatar driving method and device according to an embodiment of the present disclosure. It should be noted that Figure 1 is only an example of a system architecture to which embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure cannot be used in other applications. Device, system, environment or scenario.

As shown in Figure 1, the system architecture 100 according to this embodiment may include clients 101, 102, 103, a network 104 and a server 105. Network 104 is the medium used to provide communication links between clients 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

Users may use clients 101, 102, 103 to interact with server 105 over network 104 to receive or send messages, etc. Various communication client applications can be installed on the clients 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

The clients 101, 102, and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, and the like. The clients 101, 102, and 103 of the embodiments of the present disclosure may, for example, run applications.

The server 105 may be a server that provides various services, such as a backend management server that provides support for websites browsed by users using the clients 101, 102, and 103 (example only). The background management server can analyze and process the received user request and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the client. In addition, the server 105 can also be a cloud server, that is, the server 105 has a cloud computing function.

It should be noted that the avatar driving method provided by the embodiment of the present disclosure can be executed by the server 105 . Correspondingly, the avatar driving device provided by the embodiment of the present disclosure can be provided in the server 105. The virtual image driving method provided by the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the clients 101, 102, 103 and/or the server 105. Correspondingly, the avatar driving device provided by the embodiment of the present disclosure can also be provided in a server or server cluster that is different from the server 105 and can communicate with the clients 101, 102, 103 and/or the server 105.

In one example, the server 105 can obtain input data streams from the clients 101, 102, and 103 through the network 104, and drive the avatar based on the input data streams.

It should be understood that the number of clients, networks and servers in Figure 1 is only illustrative. You can have any number of clients, networks, and servers depending on your implementation needs.

It should be noted that in the technical solution of this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are in compliance with relevant laws and regulations and do not violate public order and good customs.

In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

Embodiments of the present disclosure provide an avatar driving method. The avatar driving method according to an exemplary embodiment of the present disclosure will be described below in conjunction with the system architecture of FIG. 1 and with reference to FIGS. 2A to 5 . The avatar driving method according to the embodiment of the present disclosure may be executed by the server 105 shown in FIG. 1 , for example.

FIG. 2A schematically shows a flow chart of an avatar driving method according to an embodiment of the present disclosure.

As shown in FIG. 2A , the avatar driving method 200 according to the embodiment of the present disclosure may include operations S210 to S250, for example.

In operation S210, an input data stream is received.

The input data stream includes multiple timing frames, and any timing frame is associated with a facial posture. The facial posture includes a first part posture and a second part posture. The second part posture includes a plurality of second sub-part postures, and any multiple first part postures. The correlation between part poses is smaller than the correlation between multiple second sub-part poses.

The correlation between any multiple first part poses is smaller than the correlation between multiple second sub-part poses. It can be understood that the correlation between any multiple first part poses is smaller than the correlation associated with any second part pose. Correlation between poses of multiple second subparts.

For example, the input data stream can be obtained by scanning a real person object's continuous three-dimensional facial expressions in the time dimension based on a camera array, and regularizing the scanned three-dimensional facial areas, so that the input data stream can reflect the time-series frames in the time dimension. , for any target timing frame, it can also reflect the vertex position of the primitive displayed in the target timing frame. The vertex position of the primitive represents the position of the vertex of the primitive. The vertex of the primitive can be understood as the vertex of the primitive (graphical element), which can be expressed by Geometric vertices are combined into primitives. The primitives can include points, line segments, polygons, etc., for example. The target object displayed by each image can be composed of multiple primitives. The vertex position of the primitive includes, for example, x coordinates, y coordinates, and z coordinates. 3D data.

For example, the first part pose and the second part pose may be obtained by dividing the face into parts in advance.

The first part posture may also include one. In this case, the first part posture may be considered to be more independent than the plurality of second sub-part postures of the second part posture.

For example, the eye area, nose area and chin area of the face can be used as the first part area, and for the facial posture displayed in any target timing frame, the first part posture corresponds to the posture of the first part area. The first part posture may include, for example, an eye area posture, a nose area posture, and a chin area posture.

For example, the chin-tooth area, the chin-lip area, and the eye-eyebrow area can also be used as the second area. For the facial posture displayed in any target timing frame, the second part posture corresponds to the posture of the second part area. . The second part posture may include, for example, a chin-tooth region posture, a chin-lip region posture, and an eye-brow region posture.

For the second part posture of the chin-tooth region posture, the corresponding second sub-part posture includes the chin sub-region posture and the tooth sub-region posture. For the second part posture of the chin-lip region posture, the corresponding second sub-part posture includes the chin sub-region posture and the lip sub-region posture. For the second part posture of the eye-eyebrow region posture, the corresponding second sub-part posture includes the eye sub-region posture and the eyebrow sub-region posture.

As shown in FIG. 2B , multiple specific examples of chin-lip region poses are schematically shown.

It should be noted that if the correlation between any multiple first part postures is less than the correlation between the second sub-part postures associated with any second part posture, it can be understood that when making the corresponding expression, the first part posture The degree of correlation between them is low, but the degree of correlation between the second sub-part postures associated with any second part posture is high. For example, in the above example, for the two first part postures corresponding to the eye area and chin area, since the eye movement will not be associated with the synchronous movement of the chin, the expression changes caused by the eye movement should be reflected in the posture changes in the eye area and chin The regional attitude should not change. Therefore, the eye region posture and the chin region posture should have low correlation or low coupling, for example, each first part posture can be decoupled.

For example, in the above example, the second region posture, the chin-tooth region posture, involves the chin region and the tooth region. In the actual human body structure, the jaw and teeth are integrally connected, so changes in expression caused by jaw movement are actually related to the synchronous movement of the teeth. Therefore, for the second part posture of the chin-tooth region posture, the associated chin region posture and tooth region posture should have strong correlation or strong coupling.

In operation S220, for any target timing frame among the plurality of timing frames, the first posture transformation coefficient is determined based on the initial reference posture and the first part posture.

For example, the initial reference pose may be the facial pose of a certain timing frame of the input data stream.

The pose transformation coefficient can be understood as the coefficient value that can be reconstructed into the corresponding facial pose. In the embodiment of the present disclosure, the first posture transformation coefficient, the second posture transformation coefficient, etc. are only used to distinguish different posture transformation coefficients.

For example, the posture transformation coefficient may include blendshape, that is, a blend deformation coefficient.

By way of example, the initial reference pose may be a "no expression" facial pose, for example.

In operation S230, the initial reference posture is updated according to the first posture transformation coefficient to obtain an updated reference posture.

In operation S240, the second posture transformation coefficient is determined according to the updated reference posture and the second part posture.

In operation S250, the avatar is driven according to the second posture transformation coefficient.

According to the avatar driving method according to the embodiment of the present disclosure, by receiving the input data stream and the first posture transformation coefficient determined based on the initial reference posture and the first part posture for any one of the target timing frames among the plurality of timing frames, it can be achieved The posture corresponding to the first part region in the initial reference posture is updated, and the updated reference posture is obtained by updating the initial reference posture according to the first posture transformation coefficient. The updated posture of the first part area can be used as the new reference posture. Updating the reference posture and the second posture transformation coefficient determined by the second part's posture can update the second part's posture based on the first part's posture update, so that the facial expression of the virtual image driven by this is more realistic.

According to the avatar driving method of the embodiment of the present disclosure, facial gestures can be subdivided based on correlation, because the correlation between different parts will affect the authenticity of facial expressions (for example, some embodiments can use The facial posture reconstructed by the transformation coefficient will show that the jaw moves to open the mouth, but the teeth do not move, which conflicts with the expression made by the real human body structure). Therefore, the avatar driving method according to the embodiment of the present disclosure can be based on low The first part posture of correlation and the second sub-part posture of high correlation can reduce the confusion of facial expressions caused by the avatar driver, making the expressions driven by the avatar more realistic. The authenticity is reflected in the low correlation, for example. The first area will not be correlated with each other to make facial expressions, and the second area with high correlation will not be able to make facial expressions independently.

FIG. 3 schematically shows a schematic diagram of an avatar driving method according to another embodiment of the present disclosure.

As shown in FIG. 3 , the avatar driving method 300 according to another embodiment of the present disclosure can, for example, use the following embodiments to implement a specific example of driving the avatar according to the second posture transformation coefficient of operation S350.

The input data stream includes vertex stream data, and each temporal frame of the vertex stream data represents the facial posture by multiple primitive vertex positions. In the example of FIG. 3 , it is schematically shown that the input data stream Input is vertex stream data. For example, the target timing frame Pi of the vertex stream data represents the corresponding facial pose fp-i by multiple primitive vertex positions 307 .

In operation S351, for the target temporal frame Pi, the second posture transformation difference data 308 is determined according to the second posture transformation coefficient 306 and the primitive vertex position 307 corresponding to the target temporal frame Pi.

For any target timing frame Pi, the corresponding primitive vertex position 307 is the real facial posture captured by the camera. The second posture transformation coefficient is used to reconstruct the facial posture. The facial posture reconstructed according to the second posture transformation coefficient is consistent with the real facial posture. If there is an error between facial poses (the real facial pose can be represented by the corresponding primitive vertex position), the second pose transformation difference data can be used to characterize the degree of difference between the facial pose reconstructed by the second pose transformation coefficient and the real facial pose.

In operation S352, a third posture transformation coefficient 309 is determined based on the second posture transformation difference data 308 and the second posture transformation coefficient 306.

In operation S353, the avatar 310 is driven according to the third posture transformation coefficient 309.

In the case where the second posture transformation difference data represents the degree of difference between the facial posture reconstructed by the second posture transformation coefficient and the real facial posture, the third posture transformation coefficient can be determined according to the second transformation difference data and the second posture transformation coefficient. The error of the third posture transformation coefficient for reconstructing the facial posture is smaller than that of the second posture transformation coefficient.

According to the avatar driving method of the embodiment of the present disclosure, by aiming at the target timing frame, the determined second posture transformation difference data can measure the reconstruction of the second posture transformation coefficient according to the second posture transformation coefficient and the primitive vertex position corresponding to the target timing frame. facial pose error. By using the second posture transformation difference data and the second posture transformation coefficient, the second posture transformation coefficient can be adjusted based on the error, and the error of the obtained third posture transformation coefficient is smaller. The avatar driven according to the third posture transformation coefficient can Make more accurate facial expressions.

In the example of FIG. 3 , operations S310 to S340 are also schematically shown. Operations S310 to S340 are respectively similar to operations S210 to S240 of the above embodiment.

For example, in the example of FIG. 3 , the receiving input data stream Input of operation S310 is schematically shown. The input data stream Input includes, for example, a total of N timing frames from timing frame P ₁ to timing frame PN, and each timing frame is associated with a corresponding facial gesture. For example, temporal frame P ₁ is associated with facial pose fp-1. Taking the target temporal frame Pi as an example, the associated facial pose fp-i includes a first part pose 301 and a second part pose 302.

In the example of FIG. 3 , it is schematically shown that operation S320 determines the first posture transformation coefficient 304 according to the initial reference posture 303 and the first part posture 301 . It is also schematically shown that S330 updates the initial reference posture 303 according to the first posture transformation coefficient 304 to obtain the updated reference posture 305. Determining the second posture transformation coefficient 306 according to the updated reference posture 305 and the second part posture 302 in S340 is also schematically shown.

FIG. 4 schematically shows a schematic diagram of determining the third posture transformation coefficient of the avatar driving method according to yet another embodiment of the present disclosure.

As shown in FIG. 4 , according to the avatar driving method according to another embodiment of the present disclosure, for example, the following embodiment can be used to implement a specific example of determining the third posture transformation coefficient according to the second posture transformation difference data and the second posture transformation coefficient.

In operation S461, a first difference posture 403 is determined based on the second posture transformation difference data 401 and the initial reference posture 402, and the first difference posture is represented by the position 404 of the corresponding primitive vertex.

The second posture transformation difference data serves as the error type data, and the first difference posture determined based on the second posture transformation difference data and the initial reference posture may present the second posture transformation difference data of the error type in the form of a facial posture.

In operation S462, the first differential posture 403 is decomposed using the first skeleton node data 405 to obtain the first skeleton-vertex associated data 406.

The first skeletal node data 405 includes the number of first skeletal nodes 405-M and the pose 405-Pos of the first skeletal node. The first bone-vertex associated data represents a graph corresponding to any first skeletal node and the first difference pose. The association weight between meta-vertices.

The corresponding primitive vertex of the first difference posture can be understood as the corresponding primitive vertex when the position of the primitive vertex is used to represent the first difference posture.

Each bone node can be associated with multiple primitive vertices corresponding to the first difference posture. Each associated primitive vertex and bone node corresponding to the first difference posture have corresponding association weights, that is, the first bone node and There is a one-to-many association between the primitive vertices corresponding to the first difference posture, and the posture of the first skeletal node has 3 degrees of freedom of movement on the x-axis, y-axis, and z-axis and 3 degrees of freedom of rotation, a total of 6 Degrees of freedom, and the vertex position of the primitive has three degrees of freedom of movement in the x-axis, y-axis, and z-axis. Compared with the position of the primitive vertex corresponding to the first difference posture, the first skeletal node is used to perform the first difference posture Bone decomposition to obtain the first bone-vertex associated data has better expressivity, can fit more complex and subtle facial postures, and is adapted to the subtle deformation of the first difference posture (the first difference posture is based on the error type The second posture difference data is obtained, so the first difference posture is a relatively subtle deformation). In addition, some complex facial gestures are difficult for real humans to make. Through the virtual image driving method of the embodiment of the present disclosure, for example, the virtual image can be driven to make facial gestures that are difficult for real humans to make.

For example, the number and posture of the first skeletal nodes can be manually set by relevant personnel, or can be automatically inferred based on the principle of skeletal decomposition according to the posture (deformation) corresponding to the first difference posture.

In the case where the first difference pose presents second pose transformation difference data of the error type in the form of a facial pose, for example, the number of first skeletal nodes and Posture.

For example, the first difference posture may be decomposed into bones based on a linear skinning decomposition algorithm, which is referred to as SSDR (Smooth Skinning Decomposition With Rigid Bones). The SSDR model can decompose linear skinning data from a series of actions, and then approximately fit the deformation corresponding to the action through a certain number of bone and vertex weight maps.

In operation S463, according to the first bone-vertex correlation data 406, the second difference posture 407 corresponding to the first bone-node correlation data is determined.

In operation S464, a third posture transformation coefficient 409 is determined based on the second posture transformation coefficient 408 and the second difference posture 407.

It should be noted that the reconstruction results of the second difference pose and the first difference pose are the same. The difference between the two is that the first difference pose is represented by the corresponding primitive vertex position, and the second difference pose is represented by the skeleton- The node is represented by the first associated data.

When it is necessary to determine the third posture transformation coefficient for reconstructing the facial posture, the second difference posture can be used to introduce the first bone-node associated data. The first bone-node associated data has stronger expressive ability and the data volume is compared to Primitive vertex positions have less data and can accommodate more subtle and larger facial poses and deformations.

Illustratively, according to the avatar driving method according to another embodiment of the present disclosure, for example, the following embodiment can also be used to implement a specific example of driving the avatar according to the third posture transformation coefficient: for the target timing frame, according to the third posture transformation coefficient and The vertex position of the primitive corresponding to the target timing frame determines the third attitude transformation difference data. When the third posture transformation difference data does not satisfy the posture transformation difference condition, the fourth posture transformation coefficient is determined based on the third posture transformation difference data and the third posture transformation coefficient, where the facial posture corresponding to the fourth posture transformation coefficient is the same as the facial posture corresponding to the fourth posture transformation coefficient. The numerical difference between the primitive vertex positions corresponding to the target timing frame satisfies the attitude transformation difference condition. The virtual image is driven according to the fourth posture transformation coefficient.

According to the avatar driving method of the embodiment of the present disclosure, in order to obtain a more accurate posture transformation coefficient, the posture change coefficient can be controlled within a specified error range through the posture transformation difference condition. When the third posture transformation coefficient determined in the above embodiment does not satisfy the posture transformation difference condition, the third posture transformation coefficient can be continuously adjusted based on the third posture transformation difference data determined by the third posture transformation coefficient until the fourth posture is obtained. The transformation coefficient satisfies the posture transformation difference condition, so that driving the virtual image according to the fourth posture transformation coefficient has higher accuracy.

Illustratively, according to the avatar driving method according to another embodiment of the present disclosure, for example, the following embodiment can be used to implement a specific example of determining the fourth posture transformation coefficient according to the third posture transformation difference data and the third posture transformation coefficient: according to the third posture transformation difference data and the third posture transformation coefficient. The three-posture transformation difference data and the initial reference posture are determined to determine the third difference posture. Use the second skeletal node data to perform skeletal decomposition on the third difference posture to obtain the second bone-vertex associated data. According to the second bone-vertex related data, a fourth difference posture corresponding to the second bone-node related data is determined. The fourth posture transformation coefficient is determined based on the third posture transformation coefficient and the fourth difference posture.

The third difference pose is characterized by the position of the corresponding primitive vertex.

The second skeletal node data includes the number and posture of the second skeletal node, and the second bone-vertex association data represents the association weight between any second skeletal node and the primitive vertex corresponding to the third difference posture.

It should be noted that, on the basis of first determining the second posture difference data in the above embodiment, the third posture transformation difference data further determined corresponds to more subtle and larger amounts of facial postures and facial gestures than the second posture difference data. deformation. According to the avatar driving method of the embodiment of the present disclosure, iteration through bone decomposition can cope with subtle and large-scale facial postures and deformations on the basis of improving accuracy. The specific principle is as explained in the above embodiment and will not be discussed here. Repeat.

Exemplarily, according to the avatar driving method according to another embodiment of the present disclosure, the posture transformation coefficient is obtained according to the posture transformation function. The posture transformation function is related to the position of the primitive vertex and the reference posture. The reference posture includes the initial reference posture, the updated At least one of the base poses. The posture transformation coefficient includes at least one of a first posture transformation coefficient, a second posture transformation coefficient, a third posture transformation coefficient and a fourth posture transformation coefficient. The third posture transformation coefficient is obtained based on the second posture transformation coefficient, and the fourth posture transformation coefficient is obtained based on the third posture transformation coefficient.

It should be noted that the “first posture transformation coefficient”, “second posture transformation coefficient”, “third posture transformation coefficient” and “fourth posture transformation coefficient” here are respectively the same as the “first posture transformation coefficient” in the above embodiment. ”, “second posture transformation coefficient”, “third posture transformation coefficient” and “fourth posture transformation coefficient” refer to the same.

For example, the posture transformation function can be characterized by the following formula (1).

C(x)＝basic+B·x (1)

x represents the value of the attitude coefficient corresponding to the vertex of the primitive, and the value of x can be between 0 and 1, for example. basic represents the base posture, such as the initial base posture or the updated base posture in the above embodiment. B represents the position of the corresponding primitive vertex.

For example, the vertex position of the primitive corresponding to the target timing frame can be regarded as B, and the initial reference posture can be regarded as basic. Using formula (1), the first posture transformation coefficient x for the target timing frame and the face corresponding to the first posture transformation coefficient can be obtained Attitude C(x).

For example, the primitive vertex position of the facial pose C(x) corresponding to the first pose transformation coefficient can be used as B, and the base pose can be updated as basic. Using formula (1), the second pose transformation coefficient x for the target timing frame can be obtained and the facial pose C(x) corresponding to the second pose transformation coefficient.

Exemplarily, according to the avatar driving method of another embodiment of the present disclosure, the posture transformation difference data is obtained according to the posture transformation difference function, and the posture transformation difference function is related to the posture transformation coefficient and the position of the key vertex; the key vertex is obtained according to the representation The properties are determined from all primitive vertices; the posture transformation difference data includes at least one of the second posture transformation difference data and the third posture transformation difference data. The third posture transformation difference data is obtained based on the third posture transformation coefficient.

It should be noted that the "posture transformation coefficient" and "third posture transformation difference data" here are respectively the same as the "posture transformation coefficient" and "third posture transformation difference data" in the above embodiment.

For example, the attitude transformation difference function can be characterized by the following formula (2).

It should be noted that the value of x in formula (1) can change, that is, the value of x can change. Each specific value of x can correspond to the relevant facial posture. Formula (2) can be used to calculate the error The condition limits the value of x and can determine an accurate specific value of x.

In formula (2), i represents any target timing frame, b _i represents the key vertex, and Xpre represents the attitude transformation coefficient calculated from the previous frame of the current frame. Formula (2) can be used for the entire input data stream, α _i b _i , β ₁ and β ₂ all represent coefficients.

To sum up, according to the avatar driving method according to the embodiment of the present disclosure, through, for example, staged multiple vertex flow calculations based on the first part posture, the second part posture, the second posture transformation difference data, the third posture transformation difference data, etc. , for example, the posture of the first part, the posture of the second part, etc. can be decoupled from each other. Decoupling can be understood as dividing the original once-executed facial posture reconstruction process into multiple times, and each time the phased posture transformation coefficient is obtained. In improving While avatar driving is accurate, it also facilitates migration between different portraits. The avatar driving method of the embodiment of the present disclosure has higher avatar driving efficiency and can be applied to a wide range of scenarios, such as virtual anchors, virtual customer service, virtual Scenes such as teachers.

FIG. 5 schematically shows a block diagram of an avatar driving device according to an embodiment of the present disclosure.

As shown in Figure 5, the avatar driving device 500 in the embodiment of the present disclosure includes, for example, an input data stream receiving module 510, a first posture transformation coefficient determination module 520, an updated reference posture determination module 530, a second posture transformation coefficient determination module 540, and Avatar driver module 540.

The input data stream receiving module 510 is configured to receive an input data stream, where the input data stream includes a plurality of time series frames, any one of the time series frames is associated with a facial posture, the facial posture includes a first part posture and a second part posture, and a second part posture. The part posture includes a plurality of second sub-part postures, and the correlation between any plurality of first part postures is smaller than the correlation between the plurality of second sub-part postures.

The first posture transformation coefficient determination module 520 is configured to determine the first posture transformation coefficient according to the initial reference posture and the first part posture for any target timing frame among the plurality of timing frames.

The updated reference posture determination module 530 is used to update the initial reference posture according to the first posture transformation coefficient to obtain the updated reference posture.

The second posture transformation coefficient determination module 540 is used to determine the second posture transformation coefficient according to the updated reference posture and the second part posture.

The virtual image driving module 550 is used to drive the virtual image according to the second posture transformation coefficient.

According to the embodiment of the present disclosure, the input data stream includes vertex stream data, and each temporal frame of the vertex stream data represents the facial posture by multiple primitive vertex positions; the avatar driving module includes: a second posture transformation difference data determination sub-module, using For the target timing frame, determine the second posture transformation difference data according to the second posture transformation coefficient and the primitive vertex position corresponding to the target timing frame; the third posture transformation coefficient determination submodule is used to transform the difference data according to the second posture and The second posture transformation coefficient determines the third posture transformation coefficient; the virtual image driving submodule is used to drive the virtual image according to the third posture transformation coefficient.

According to an embodiment of the present disclosure, the third posture transformation coefficient determination sub-module includes: a first difference posture determination unit, configured to determine the first difference posture according to the second posture transformation difference data and the initial reference posture, and the first difference posture is determined by the corresponding The position representation of the vertex of the primitive; the bone-vertex first associated data determination unit is used to perform skeletal decomposition of the first difference posture using the first bone node data to obtain the first bone-vertex associated data, wherein the first bone node data Including the number and posture of the first skeletal nodes, the first bone-vertex associated data represents the association weight between any first skeletal node and the primitive vertex corresponding to the first difference posture; the second difference posture determination unit is used for Determine the second difference posture corresponding to the first bone-node correlation data according to the first bone-vertex correlation data; the third posture transformation coefficient determination unit is used to determine the third posture transformation coefficient according to the second posture transformation coefficient and the second difference posture. Attitude transformation coefficient.

According to an embodiment of the present disclosure, the avatar driving submodule includes: a third posture transformation difference data determination unit, configured to determine the third posture for the target timing frame according to the third posture transformation coefficient and the primitive vertex position corresponding to the target timing frame. transformation difference data; a fourth posture transformation coefficient determination unit, configured to determine the fourth posture transformation based on the third posture transformation difference data and the third posture transformation coefficient when the third posture transformation difference data does not satisfy the posture transformation difference condition. coefficient, wherein the difference value between the facial posture corresponding to the fourth posture transformation coefficient and the primitive vertex position corresponding to the target timing frame satisfies the posture transformation difference condition; the avatar driving unit is used to drive the avatar according to the fourth posture transformation coefficient .

According to an embodiment of the present disclosure, the fourth posture transformation coefficient determination unit includes: a third difference posture determination subunit, configured to determine the third difference posture according to the third posture transformation difference data and the initial reference posture, and the third difference posture is determined by the corresponding The position representation of the primitive vertices; the bone-vertex second associated data determination subunit is used to use the second bone node data to perform bone decomposition on the third difference posture to obtain the bone-vertex second associated data, where the second bone node The data includes the number and posture of the second skeletal node. The second bone-vertex association data represents the association weight between any second skeletal node and the primitive vertex corresponding to the third difference posture; the fourth difference posture determination subunit, Used to determine the fourth difference posture corresponding to the second bone-node related data according to the second bone-vertex related data; the fourth posture transformation coefficient determination subunit is used to determine the fourth difference posture based on the third posture transformation coefficient and the fourth difference posture. Determine the fourth posture transformation coefficient.

According to an embodiment of the present disclosure, the first part posture includes at least one of the following: eye area posture, nose area posture, and chin area posture; the second part posture includes at least one of the following: chin-tooth area posture, chin-lip area Posture, eye-eyebrow area posture, chin-tooth area posture includes chin sub-region posture and tooth sub-region posture, chin-lip area posture includes chin sub-region posture and lip sub-region posture, eye-eyebrow area posture includes eye sub-region posture and eyebrow area posture.

According to the embodiment of the present disclosure, the posture transformation coefficient is obtained according to the posture transformation function. The posture transformation function is related to the reference posture and the vertex position of the primitive. The reference posture includes the initial reference posture and the updated reference posture; the posture transformation coefficient includes the first posture transformation coefficient. , at least one of the second posture transformation coefficient, the third posture transformation coefficient and the fourth posture transformation coefficient. The third posture transformation coefficient is obtained based on the second posture transformation coefficient, and the fourth posture transformation coefficient is obtained based on the third posture transformation coefficient.

According to embodiments of the present disclosure, the posture transformation difference data is obtained based on the posture transformation difference function, which is related to the posture transformation coefficient and the position of the key vertex; the key vertex is determined from the full amount of primitive vertices based on representation; The posture transformation difference data includes at least one of the second posture transformation difference data and the third posture transformation difference data. The third posture transformation difference data is obtained based on the third posture transformation coefficient.

It should be understood that the embodiments of the device part of the disclosure are the same or similar to the embodiments of the method part of the disclosure, and the technical problems solved and the technical effects achieved are also the same or similar, and the disclosure will not be repeated here.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

Figure 6 shows a schematic block diagram of an example electronic device 600 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are examples only and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 6 , the device 600 includes a computing unit 601 that can execute according to a computer program stored in a read-only memory (ROM) 602 or loaded from a storage unit 608 into a random access memory (RAM) 603 Various appropriate actions and treatments. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. Computing unit 601, ROM 602 and RAM 603 are connected to each other via bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

Multiple components in device 600 are connected to I/O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as magnetic disk, optical disk, etc. ; and communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices through computer networks such as the Internet and/or various telecommunications networks.

Computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the avatar driving method. For example, in some embodiments, the avatar driving method may be implemented as a computer software program that is tangibly included in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 600 via ROM 602 and/or communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the avatar driving method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the avatar driving method in any other suitable manner (eg, by means of firmware).

Various implementations of the systems and techniques described above may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip implemented in a system (SOC), complex programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include implementation in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor The processor, which may be a special purpose or general purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device. An output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that the program codes, when executed by the processor or controller, cause the functions specified in the flowcharts and/or block diagrams/ The operation is implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, laptop disks, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and may be provided in any form, including Acoustic input, voice input or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., A user's computer having a graphical user interface or web browser through which the user can interact with implementations of the systems and technologies described herein), or including such backend components, middleware components, or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communications network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

Computer systems may include clients and servers. Clients and servers are generally remote from each other and typically interact over a communications network. The relationship of client and server is created by computer programs running on corresponding computers and having a client-server relationship with each other.

It should be understood that various forms of the process shown above may be used, with steps reordered, added or deleted. For example, each step described in the present disclosure can be executed in parallel, sequentially, or in a different order. As long as the desired results of the technical solution disclosed in the present disclosure can be achieved, there is no limitation here.

The above-mentioned specific embodiments do not constitute a limitation on the scope of the present disclosure. It will be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions are possible depending on design requirements and other factors. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this disclosure shall be included in the protection scope of this disclosure.

Claims (16)

1. An avatar driving method, comprising:

receiving an input data stream, wherein the input data stream comprises a plurality of time-sequence frames, any one of the time-sequence frames is associated with a facial pose, the facial pose comprises a first part pose and a second part pose, the second part pose comprises a plurality of second sub-part poses, and the correlation between any plurality of the first part poses is smaller than the correlation between the plurality of the second sub-part poses;

determining a first posture transformation coefficient according to an initial reference posture and the first position posture for any one target time sequence frame in the plurality of time sequence frames;

updating the initial reference gesture according to the first gesture transformation coefficient to obtain an updated reference gesture;

determining a second posture transformation coefficient according to the updated reference posture and the second position posture; and

Driving the avatar according to the second pose transform coefficient;

wherein the input data stream comprises vertex stream data, each of the time-series frames of the vertex stream data characterizing the facial pose by a plurality of primitive vertex positions; the driving the avatar according to the second pose transform coefficient includes:

determining second posture transformation difference data according to the second posture transformation coefficient and the primitive vertex position corresponding to the target time sequence frame aiming at the target time sequence frame;

determining a third attitude transformation coefficient according to the second attitude transformation difference data and the second attitude transformation coefficient; and

and driving the avatar according to the third posture transformation coefficient.

2. The method of claim 1, wherein the determining a third pose transform coefficient from the second pose transform difference data and the second pose transform coefficient comprises:

determining a first differential gesture according to the second gesture transformation differential data and the initial reference gesture, wherein the first differential gesture is represented by the position of the vertex of the corresponding primitive;

performing skeleton decomposition on the first difference gesture by using first skeleton node data to obtain first association data of skeleton-vertex, wherein the first skeleton node data comprises the number and the position of first skeleton nodes, and the first association data of the skeleton-vertex represents association weights between any one first skeleton node and the primitive vertex corresponding to the first difference gesture;

Determining a second difference gesture corresponding to the bone-vertex first association data according to the bone-vertex first association data; and

and determining the third posture transformation coefficient according to the second posture transformation coefficient and the second difference posture.

3. The method of claim 1, wherein the driving the avatar according to the third pose transform coefficient comprises:

determining third posture transformation difference data according to the third posture transformation coefficient and the primitive vertex position corresponding to the target time sequence frame aiming at the target time sequence frame;

determining a fourth posture transformation coefficient according to the third posture transformation difference data and the third posture transformation coefficient under the condition that the third posture transformation difference data does not meet the posture transformation difference condition, wherein a difference value between a face posture corresponding to the fourth posture transformation coefficient and a primitive vertex position corresponding to the target time sequence frame meets the posture transformation difference condition; and

and driving the avatar according to the fourth posture transformation coefficient.

4. A method according to claim 3, wherein said determining a fourth pose transform coefficient from said third pose transform difference data and said third pose transform coefficient comprises:

Determining a third difference gesture according to the third gesture transformation difference data and the initial reference gesture, wherein the third difference gesture is represented by the position of the vertex of the corresponding primitive;

performing skeleton decomposition on the third differential gesture by using second skeleton node data to obtain skeleton-vertex second association data, wherein the second skeleton node data comprises the number and the positions of second skeleton nodes, and the skeleton-vertex second association data represents association weights between any one of the second skeleton nodes and primitive vertices corresponding to the third differential gesture;

determining a fourth difference gesture corresponding to the bone-vertex second association data according to the bone-vertex second association data; and

and determining the fourth posture transformation coefficient according to the third posture transformation coefficient and the fourth difference posture.

5. The method of any of claims 1-4, wherein the first site pose comprises at least one of: eye region pose, nose region pose, and chin region pose; the second site pose includes at least one of: a chin-tooth region pose, a chin-lip region pose, an eye-eyebrow region pose, the chin-tooth region pose comprising a chin sub-region pose and a tooth sub-region pose, the chin-lip region pose comprising a chin sub-region pose and a lip sub-region pose, the eye-eyebrow region pose comprising an eye sub-region pose and an eyebrow sub-region pose.

6. The method of any of claims 1-4, wherein the pose transform coefficients are derived from a pose transform function, the pose transform function being related to a reference pose, primitive vertex positions, the reference pose comprising the initial reference pose, the updated reference pose; the posture transformation coefficients comprise at least one of the first posture transformation coefficient, the second posture transformation coefficient, a third posture transformation coefficient and a fourth posture transformation coefficient, wherein the third posture transformation coefficient is obtained according to the second posture transformation coefficient, and the fourth posture transformation coefficient is obtained according to the third posture transformation coefficient.

7. The method of claim 1, wherein the pose transformation difference data is derived from a pose transformation difference function that is related to the pose transformation coefficients and the positions of the key vertices; the key vertexes are determined from the primitive vertexes of the whole quantity according to the characterizations; the posture transformation difference data comprises at least one of the second posture transformation difference data and third posture transformation difference data, wherein the third posture transformation difference data is obtained according to the third posture transformation coefficient.

8. An avatar driving apparatus, comprising:

an input data stream receiving module configured to receive an input data stream, where the input data stream includes a plurality of timing frames, any one of the timing frames being associated with a facial pose, the facial pose including a first part pose and a second part pose, the second part pose including a plurality of second sub-part poses, a correlation between any plurality of the first part poses being less than a correlation between a plurality of the second sub-part poses;

the first posture transformation coefficient determining module is used for determining a first posture transformation coefficient according to the initial reference posture and the first position posture aiming at any one target time sequence frame in the plurality of time sequence frames;

the updating reference posture determining module is used for updating the initial reference posture according to the first posture transformation coefficient to obtain an updated reference posture;

the second posture transformation coefficient determining module is used for determining a second posture transformation coefficient according to the updated reference posture and the second position posture; and

an avatar driving module for driving an avatar according to the second pose conversion coefficient;

wherein the input data stream comprises vertex stream data, each of the time-series frames of the vertex stream data characterizing the facial pose by a plurality of primitive vertex positions; the avatar driving module includes:

The second gesture transformation difference data determining submodule is used for determining second gesture transformation difference data according to the second gesture transformation coefficient and the primitive vertex position corresponding to the target time sequence frame aiming at the target time sequence frame;

a third pose transform coefficient determination sub-module configured to determine a third pose transform coefficient according to the second pose transform difference data and the second pose transform coefficient;

and the avatar driving sub-module is used for driving the avatar according to the third gesture transformation coefficient.

9. The apparatus of claim 8, wherein the third pose transform coefficient determination submodule comprises:

a first difference gesture determining unit, configured to determine a first difference gesture according to the second gesture transformation difference data and the initial reference gesture, where the first difference gesture is represented by a position of a vertex of a corresponding primitive;

a bone-vertex first association data determining unit, configured to perform bone decomposition on the first differential gesture by using first bone node data to obtain bone-vertex first association data, where the first bone node data includes the number and the pose of first bone nodes, and the bone-vertex first association data characterizes an association weight between any one of the first bone nodes and a primitive vertex corresponding to the first differential gesture;

A second differential posture determining unit configured to determine a second differential posture corresponding to the bone-vertex first association data according to the bone-vertex first association data; and

and a third posture transformation coefficient determining unit configured to determine the third posture transformation coefficient according to the second posture transformation coefficient and the second difference posture.

10. The apparatus of claim 8, wherein the avatar driving sub-module comprises:

a third gesture transformation difference data determining unit, configured to determine, for the target time-series frame, third gesture transformation difference data according to the third gesture transformation coefficient and the primitive vertex position corresponding to the target time-series frame;

a fourth posture transformation coefficient determining unit, configured to determine a fourth posture transformation coefficient according to the third posture transformation difference data and the third posture transformation coefficient, in a case where the third posture transformation difference data does not satisfy a posture transformation difference condition, where a difference value between a face posture corresponding to the fourth posture transformation coefficient and a primitive vertex position corresponding to the target time-series frame satisfies the posture transformation difference condition; and

And an avatar driving unit for driving the avatar according to the fourth pose conversion coefficient.

11. The apparatus of claim 10, wherein the fourth pose transform coefficient determination unit comprises:

a third difference pose determination subunit, configured to determine a third difference pose according to the third pose transformation difference data and the initial reference pose, where the third difference pose is represented by a position of a vertex of a corresponding primitive;

a bone-vertex second association data determining subunit, configured to perform bone decomposition on the third differential gesture by using second bone node data to obtain bone-vertex second association data, where the second bone node data includes the number and the pose of second bone nodes, and the bone-vertex second association data characterizes an association weight between any one of the second bone nodes and a primitive vertex corresponding to the third differential gesture;

a fourth differential gesture determining subunit, configured to determine a fourth differential gesture corresponding to the second association data of the bone-vertex according to the second association data of the bone-vertex; and

a fourth posture-transformation-coefficient determining subunit configured to determine the fourth posture transformation coefficient according to the third posture transformation coefficient and the fourth differential posture.

12. The apparatus of any of claims 8-11, wherein the first site pose comprises at least one of: eye region pose, nose region pose, and chin region pose; the second site pose includes at least one of: a chin-tooth region pose, a chin-lip region pose, an eye-eyebrow region pose, the chin-tooth region pose comprising a chin sub-region pose and a tooth sub-region pose, the chin-lip region pose comprising a chin sub-region pose and a lip sub-region pose, the eye-eyebrow region pose comprising an eye sub-region pose and an eyebrow sub-region pose.

13. The apparatus of any of claims 8-11, wherein the pose transform coefficients are derived from a pose transform function, the pose transform function being related to a reference pose, primitive vertex position, the reference pose comprising the initial reference pose, the updated reference pose; the posture transformation coefficients comprise at least one of the first posture transformation coefficient, the second posture transformation coefficient, a third posture transformation coefficient and a fourth posture transformation coefficient, wherein the third posture transformation coefficient is obtained according to the second posture transformation coefficient, and the fourth posture transformation coefficient is obtained according to the third posture transformation coefficient.

14. The apparatus of claim 8, wherein the pose transform difference data is derived from a pose transform difference function, the pose transform difference function being related to the pose transform coefficients and the locations of the key vertices; the key vertexes are determined from the primitive vertexes of the whole quantity according to the characterizations; the posture transformation difference data comprises at least one of the second posture transformation difference data and third posture transformation difference data, wherein the third posture transformation difference data is obtained according to the third posture transformation coefficient.

15. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-7.