DE112019001058T5 - VOICE EFFECTS BASED ON FACIAL EXPRESSIONS - Google Patents

Abstract

Embodiments of the present disclosure may provide techniques for customizing audio and / or video information of a video clip based at least in part on facial and / or voice characteristic characteristics extracted from hardware components. For example, in response to detecting a request to create an avatar video clip of a virtual avatar, a video signal associated with a face in a field of view of a camera and an audio signal may be recorded. Voice feature characteristics and facial feature characteristics can be extracted from the audio signal and the video signal, respectively. In some examples, in response to detecting a request to preview the avatar video clip, a customized audio signal may be generated based at least in part on the facial feature characteristics and the voice feature characteristics, and using the customized audio signal, a preview of the Video clips of the virtual avatar are displayed.

Description

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of the non-provisional U.S. Patent Application No. 15 / 908,603 , filed February 28, 2018, entitled "Voice Effects Based on Facial Expressions," and U.S. Partial Continuation Application No. 16 / 033,111 , filed July 11, 2018, entitled "Techniques for Providing Audio and Video Effects," which disclosures are hereby incorporated in their entirety by reference for all purposes.

STATE OF THE ART

Multimedia content such as B. Emojis, can be sent as part of message communications. The emojis can represent a variety of predefined people, objects, actions and / or other things. Some messaging applications allow users to choose from a predefined library of emojis that can be sent as part of a message, which may contain other content (e.g., other multimedia and / or textual content). Animojis are a type of this other multimedia content where a user can select an avatar (e.g. a doll) to represent themselves. The Animoji can move and speak as if it were a video of the user. Animojis allow users to create personalized versions of emojis in a fun and creative way.

ABSTRACT

Embodiments of the present invention can provide systems, methods, and a computer readable medium for implementing techniques for revising and playing avatar video clips. In some examples, a computing device may present a user interface (UI) for tracking a user's face and displaying a virtual avatar representation (e.g., a doll or a video character version of the user's face). After identifying a recording request, the computing device can record audio and video information, extract and capture context as well as facial feature characteristics and voice feature characteristics, revise the audio and / or video information based at least in part on the extracted / identified features and using the revised audio and / or video information represents a video clip of the avatar.

In some embodiments, a computer implemented method of implementing various audio and video effects techniques can be provided. The method may include displaying an interface for creating virtual avatars. The method can also include displaying a first preview content of a virtual avatar in the interface for creating virtual avatars, the first preview content of the virtual avatar corresponding to real-time preview video frames of a user portrait recording in a field of view of the camera and associated changes in the appearance of the portrait recording. The method may also include capturing an input to the virtual avatar creation interface while displaying the first preview content of the virtual avatar. In some examples, in response to detecting the input to the virtual avatar generation interface, the method may further include: recording a video signal associated with the user portrait recording via the camera during a recording session; recording a user audio signal via the camera during the recording session, extracting audio feature characteristics from the recorded user audio signal, and extracting facial feature characteristics associated with the face from the recorded video signal. In addition, in response to detecting a course of the recording session, the method may also include: generating an adjusted audio signal from the recorded audio signal based at least in part on the facial feature characteristics and the audio feature characteristics, generating a second preview content of the virtual avatar in the interface to the Generation of virtual avatars according to the facial feature characteristics and the adapted audio signal and presentation of the second preview content in the interface for generating virtual avatars.

In some embodiments, the method may also include storing facial feature metadata associated with the facial feature characteristics extracted from the video signal and generating customized facial feature metadata from the facial feature metadata based at least in part on the facial feature characteristics. Characteristics and the audio feature characteristics. In addition, the second preview of the virtual avatar may also be displayed according to the customized face metadata. In some examples, the first preview of the virtual avatar may be displayed according to preview facial feature characteristics identified according to changes in the appearance of the face during a preview session.

In some embodiments, an electronic device for implementing various audio and video effects techniques can be provided. The system may include a camera, microphone, library of pre-recorded / predetermined audio, and one or more processors in communication with the camera and microphone. In some examples, the processors can be configured to execute computer-executable instructions to perform operations. The operations may include detecting input to a virtual avatar creation interface while displaying a first preview of a virtual avatar. The operations may also include initiating a recording session including in response to capturing input to the virtual avatar creation interface. The recording session can include: recording via the camera a video signal which belongs to a face in a field of view of the camera, recording via the microphone an audio signal which corresponds to the recorded video signal, extracting audio feature characteristics from the recorded audio signal and extracting facial features -Characteristics associated with the face from the recorded video signal. In some examples, the operations may also include generating an adjusted audio signal based at least in part on the audio feature characteristics and the facial feature characteristics, and presenting the second preview content in the virtual avatar generation interface, at least in response to the detection of a flow the recording session.

In some cases, the audio signal can also be adjusted based at least in part on a type of the virtual avatar. In addition, the type of virtual avatar can be received based at least in part on a selection option of avatar types that is displayed in the interface for generating virtual avatars. In some cases, the type of virtual avatar may include an animal species, and the customized audio signal may be generated based at least in part on a predetermined sound associated with the animal species. The use and timing of predetermined sounds can be based on audio features from the recorded audio and / or facial features from the recorded video. This predetermined sound can also be modified itself based on audio features from the recorded audio and / or facial features from the recorded video. In some examples, the one or more processors may be further configured to determine whether a portion of the audio signal corresponds to the face in the field of view. Additionally, according to a determination that the part of the audio signal corresponds to the face, the part of the audio signal may be stored for use in generating the customized audio signal and / or according to a determination that the part of the audio signal does not correspond to the face, at least the part of the audio signal discarded and not considered for modification and / or playback. Additionally, the audio feature characteristics can include features of a voice associated with the face in the field of view. In some examples, the one or more processors may be further configured to store facial feature metadata associated with the facial feature characteristics extracted from the video signal. In some examples, the one or more processors may be further configured to store audio feature metadata associated with the audio feature characteristics extracted from the audio signal. Further, the one or more processors may be further configured to generate customized face metadata based at least in part on the facial feature characteristics and the audio feature characteristics, and the second preview of the virtual avatar may be generated according to the customized face metadata and the customized audio signal .

In some embodiments, a computer readable medium can be provided. The computer readable medium may include computer executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations. The operations may include performing the following acts in response to detecting a request to create an avatar video clip of a virtual avatar: recording, via a camera of an electronic device, a video signal associated with a face in a field of view of the camera an audio signal through a microphone of the electronic device, extracting voice feature characteristics from the recorded audio signal, and extracting facial feature characteristics associated with the face from the recorded video signal. In response to detecting a request to preview the avatar video clip, the operations may also include performing the following acts: generating an adjusted audio signal based at least in part on the facial feature characteristics and the voice feature characteristics, and displaying a Preview the video clip of the virtual avatar using the customized audio signal.

In some embodiments, the audio signal may be based at least in part on a Facial expression identified in the facial feature characteristics associated with the face. In some cases, the audio signal may be adjusted based at least in part on a level, pitch, duration, format, or a change in a voice characteristic associated with the face. Further, in some embodiments, the one or more processors may be further configured to perform the operations including sending the video clip of the virtual avatar to another electronic device.

The following detailed description, together with the accompanying drawings, provides a better understanding of the nature and advantages of the present disclosure.

Figure list

• 1 Figure 13 is a simplified block diagram illustrating an example flow for providing audio and / or video effects techniques as described herein according to at least one example.
• 2 Figure 13 is another simplified block diagram illustrating an example flow for providing audio and / or video effects techniques as described herein according to at least one example.
• 3 Figure 3 is another simplified block diagram illustrating hardware and software components for providing audio and / or video effects techniques as described herein according to at least one example.
• 4th FIG. 13 is a flow chart illustrating the provision of audio and / or video effects techniques as described herein, according to at least one example.
• 5 Figure 13 is another flow chart illustrating the provision of audio and / or video effects techniques as described herein, according to at least one example.
• 6th Figure 3 is a simplified block diagram illustrating a user interface for providing audio and / or video effects techniques as described herein, according to at least one example.
• 7th Figure 13 is another flow chart illustrating the provision of audio and / or video effects techniques as described herein, according to at least one example.
• 8th Figure 13 is another flow chart illustrating the provision of audio and / or video effects techniques as described herein, according to at least one example.
• 9 Figure 13 is a simplified block diagram illustrating a computer architecture for providing audio and / or video effects techniques as described herein, according to at least one example.

DETAILED DESCRIPTION

Certain embodiments of the present disclosure relate to devices, a computer readable medium, and methods of implementing various techniques for providing voice effects (e.g., revised audio) based at least in part on facial expressions. Additionally, in some cases, the various techniques may also provide video effects based at least in part on audio characteristics of a recording. Further, the various techniques can also provide voice effects and video effects (e.g., together) based at least in part on one or both of facial expressions and audio characteristics of a recording. In some examples, the voice effects and / or the video effects may be presented in a user interface (UI) configured to display an animated film representation of a user (e.g., an avatar or a digital doll). Such an avatar representing a user can be thought of as an animoji because it can look like an emoji character that most smartphone users are familiar with; however, he can be animated to imitate actual movements of the user.

For example, a user of a computing device may be provided with a UI for generating an Animoji video (e.g., a video clip). The video clip can be limited to a predetermined length of time (e.g., 10 seconds, 30 seconds, or the like) or the video clip can be unlimited. In the UI, a preview area can present the user with a real-time representation of his face using an avatar character. Different avatar characters can be provided and a user can even be able to create or import their own avatar. The preview area can be configured to provide an initial preview of the avatar and a preview of the recorded video clip. In addition, the recorded video clip can be previewed in its original form (e.g. without any video or audio effects) or it can be previewed with audio and / or video effects. In some cases, the user can select an avatar after the initial video clip has been recorded. The video clip can then switch from one avatar to another, using the same or different video effects on it as required be applied. For example, when viewing the raw preview (e.g., the original shape with no effects) and the user changes avatar characters, the UI can be updated to show a representation of the same video clip but with the newly selected avatar. In other words, the facial features and the audio (e.g. the user's voice) recorded during the recording can be represented by each of the avatars (e.g. without any effect). In the preview, it will appear as if the avatar character is moving in the same way as the user moved during the recording and speaks what the user said during the recording.

For example, a user can select a first avatar (e.g. a head of a unicorn) via the UI or a standard avatar can initially be provided. The UI displays the avatar (in this example, the head of a cartoon unicorn if selected by the user, or any other available doll by default) in the preview area and the device begins to record audio and / or video information (e.g. using one or more microphones and / or one or more cameras). In some cases, only video information is needed for the initial preview screen. The video information can be analyzed and facial features can be extracted. These extracted facial features can then be applied to the unicorn's face in real time so that the initial preview of the unicorn's head appears to mirror that of the user. In some cases, the term “real time” is used to indicate that the results of the extraction, rendering, rendering and presentation are performed in response to the user's movement and can be rendered essentially instantaneously. It appears to the user as if they are looking in the mirror, except that the image of their face is replaced with an avatar.

Although the user's face is in the line of sight (e.g., the field of view) of a camera of the device, the UI still represents the initial preview. After selecting a recording option (e.g. a virtual button) on the UI the device begins to record a video which has an audio component. In some examples, this includes a camera capturing frames and a microphone capturing audio information. A special camera can be used, which is also able to record 3-dimensional (3D) information. Additionally, in some examples, any camera capable of capturing video can be used. The video can be saved in its original form and / or metadata associated with the video can be saved. Thus, the recording of the video and / or audio information can differ from the storage of the information. For example, taking the information may include capturing the information and at least caching it so that it is available for processing. The processed data can also be cached until it is determined whether the data should be saved or simply used. For example, during the initial preview, while the face of the user is being displayed as a doll in real time, the video data (e.g., metadata associated with the data) may be cached as it is being cast and displayed on the doll . However, this data does not need to be saved permanently, so the initial preview is not reusable or recoverable.

Alternatively, in some examples, once the user selects the UI to record, the video and audio may be saved more permanently. In this manner, the audio and video (A / V) data can be analyzed, processed, etc. to provide the audio and video effects described herein. In some examples, the video data can be processed to extract facial features (e.g., facial feature characteristics) and these facial features can be saved as metadata for the Animoji video clip. The metadata set can be stored with an identifier (ID) indicating the time, date and user associated with the video clip. In addition, the audio data can be saved with the same or a different ID. Once saved, or in some examples prior to saving, the system (e.g., processors of the device) may extract audio feature characteristics from the audio data and facial feature characteristics from the video file. This information can be used to identify context, keywords, intentions, and / or emotions of the user, and video and audio effects can be incorporated into audio and video data before the doll is displayed. In some examples, the audio signal can be adjusted to include other words, sounds, tones, voices, timings, etc. based at least in part on the extracted features. In addition, the video data (e.g. the metadata) can also be adjusted in some examples. In some examples, audio features are extracted in real time during the preview itself. These audio features can be avatar-specific and can only be generated when a preview of the associated avatar is viewed. The audio features can be avatar-independent and can be generated for all avatars. The audio signal can also be based in part on these real-time audio feature extractions and with the pre-stored extracted video features generated during or after the recording process, but before viewing the preview.

Once the video and audio data have been adjusted, based at least in part on the extracted characteristics, a second preview of the doll can be displayed. This display can be performed for each possible doll, so that when the user scrolls through different dolls and selects different dolls, the adjusted data is already presented. Or the representation can take place after the selection of the respective doll. In any case, the second preview can be displayed as soon as the user selects a doll. The second preview plays the video clip that was recorded by the user, but with the adjusted audio and / or video. In the example above, if the user recorded themselves with an annoyed tone (e.g., a harsh voice and a furrowed eyebrow), the context or intention of anger can be grasped and the audio file can be adjusted to match includes a growling sound. Thus the second preview looks like a unicorn speaking the words the user spoke; however, the user's voice can be adjusted to sound like a growl or to sound more like a baritone (e.g., lower). The user could then save the second preview or select it to be sent to another user (e.g., through a messaging application or the like). In some examples, the Animoji video clips below and above can be shared as .mov files. However, in other examples, the techniques described may be used in real time (e.g., with video messaging or the like).

1 is a simplified block diagram showing an example flow 100 illustrated for providing audio and / or video effects based at least in part on audio and / or video characteristics captured in a recording of a user. In the example process 100 there are two separate sessions: a recording session 102 and a playing session 104 . In the recording session 102 can the device 106 in block 110 record a video, which is an audio component of a user 108 having. In some examples, the video and audio may be recorded (e.g., compiled) separately using two different devices (e.g., a microphone and a camera). The recording of the video and audio may be based at least in part on a selection of a recording option by the user 108 to be triggered. In some examples, the user can 108 in block 112 speak the word "hello". In addition, the device 106 in the block 112 further record the video and / or audio components of the user's actions. In block 114 can the device 106 continue to record the video and / or audio components, and in this example the user can 108 speak the word "bark". In the block 114 can the device 106 also extract spoken words from the audio information. However, in other examples, spoken word extraction (or any extraction of audio features) may actually take place after the recording session 102 is completed. In other examples, spoken word extraction (or any extraction of audio features) can actually be performed in real time during the preview block 124 occur. It is also possible that the extraction (e.g. analyzing the audio) takes place in real time during the recording session 102 is still going on. In either case, in the avatar process initiated by the device 106 is performed, the extraction identifies that the user has spoken the word "bark" and logic can be applied to determine which audio effects to implement.

For example, the recording session 102 end when the user 108 re-selects the record option (e.g. indicating a desire to end recording), chooses a record stop option (e.g., the record option may act as a record stop option while recording), or based at least in part on the expiration of a period of time (e.g. B. 10 seconds, 30 seconds or the like). In some cases this time period can be automatically predetermined, while in other cases it can be selected by the user (e.g. can be selected from a list of options or can be entered in free form via a text input interface). Once the recording is finished, the user can 108 Select a preview option, which indicates that the user is 108 preview the recording. One option could be to play the original recording without any visual or audio effects. However, another option could be to play a revised version of the video clip. Based at least in part on the detection of the spoken word “bark”, the audio and / or video of the video clip may have been revised in the avatar process.

In block 116 can the device 106 display an avatar (also referred to as a doll and / or an animoji) 118 on a screen. The device 106 can also use a speaker 120 configured to play audio associated with the video clip. In this example, the block corresponds to 116 same point in time as the block 110 to which the user 108 could have opened his mouth but has not yet spoken. Hence the avatar can 118 be shown with his mouth open; however, is from the speaker 120 no audio shown yet. In block 122 that the block 112 corresponds to where the user 108 “Hello” can be said by the avatar 118 are represented in the avatar process with an avatar-specific voice. In other words, in the block 122 a pre-defined dog's voice can be used to say the word "hello". The dog voice word "Hello" can through the loudspeaker 120 being represented. As will be described in greater detail below, a variety of different animal avatars (and avatars of other characters) are available for selection by the user 108 available. In some examples, each avatar may have a specific pre-defined voice that best fits that avatar. For example, a dog can have a dog's voice, a cat can have a cat's voice, a pig can have a pig's voice, and a robot can have a robotic voice. These avatar-specific voices may be prerecorded or may be associated with certain frequency or audio conversions that can be done by performing mathematical operations on the original sound so that each user's voice can be converted to sound like the dog's voice . However, each user's dog voice may sound different based at least in part on the particular audio conversion performed.

In block 124 the avatar process can replace the spoken word (e.g. "bark") with an avatar-specific word. In this example, the sound of a dog barking (e.g. a recorded or simulated dog bark) can be inserted into the audio (e.g. in place of the word "bark") so that if it is played while the video clip is being displayed, from the speaker 120 a "woof" is represented. In some examples, different avatar-specific words are displayed at 124 based at least in part on the selection of different avatars, and in other examples the same avatar-specific word may be displayed regardless of the selection of the avatar. For example, if the user 108 "Bark" has spoken, a "woof" could be displayed when the dog avatar is selected. However, if in the same case the user 108 If you later selected the cat avatar for the same process, there are a few options for reworking the audio. In one example, the process could convert the "bark" into a "woof" even if a "woof" would not be appropriate for a cat. In another example, the process could convert "bark" into a recorded or simulated "meow" based at least in part on the selection of the cat avatar. And in yet another example, the process could ignore the "bark" for avatars other than the canine avatar. Thus, there may be a second stage of audio feature analysis that is performed at 114 even after the extraction. Video and audio features can also influence the processing of the avatar-specific utterances. For example, the strength and pitch of the voice and intonation with which a user pronounces “bark” can be detected as part of the audio feature extraction, and this can cause the system to select a particular “woof” sample, or such a sample before and / or convert during the preview process.

2 is another simplified block diagram showing an example flow 200 illustrated for providing audio and / or video effects based at least in part on audio and / or video characteristics captured in a recording of a user. In the example process 200 is there, much like in the sample flow 100 the 1 , two separate sessions: a recording session 202 and a playback session 204 . In the recording session 202 can be a device 206 in block 210 record a video which is an audio component of the user 208 having. The recording of the video and audio may be based at least in part on a selection of a recording option by the user 208 to be triggered. In some examples, the user can 208 in block 212 speak the word "hello". In addition, the device 206 in the block 212 further record the video and / or audio components of the user's actions. In block 214 can the device 206 continue to record the video and / or audio components, and in this example the user can 208 keep your mouth open but don't speak. In the block 214 can the device 206 also extract facial expressions from the video. However, in other examples, facial feature extraction (or any extraction of video features) may actually take place after the recording session 202 is completed. It is still possible for the extraction (e.g. analyzing the video) to be done in real time during the recording session 202 is still going on. In either case, in the avatar process initiated by the device 206 is performed, the extraction identifies that the user has briefly opened his mouth (e.g., without speaking anything) and logic can be applied to determine what audio and / or video effects to implement . In some examples, the determination that the user kept his mouth open without speaking, requiring extraction and analysis of both an audio and a video. For example, extraction of the facial feature characteristics (e.g. open mouth) may not be sufficient, and it may also be necessary that the user is detected in the process 208 did not speak during the same time period of the recording. Video and audio features can also influence the processing of the avatar-specific utterances. For example, the duration of the opening of the mouth, opening of the eyes, etc. may cause the system to select a particular "woof" sample or to convert such a sample before and / or during the preview process. Such conversion alters the strength and / or duration of the woof to match the sensed opening and closing of the user's mouth.

For example, the recording session 202 end when the user 208 re-selects the record option (e.g. indicating a desire to end recording), selects a record stop option (e.g. the record option may act as a record stop option while recording), or based at least in part on a lapse of time (e.g. B. 20 seconds, 30 seconds or the like). Once the recording is finished, the user can 208 Select a preview option, which indicates that the user is 208 preview the recording. One option could be to play the original recording without any visual or audio effects. However, another option could be to play a revised version of the recording. Based at least in part on the detection of the facial expression (e.g., the open mouth), the audio and / or video of the video clip may have been revised by the avatar process.

In block 216 can the device 206 an avatar (also known as a doll and / or an animoji) 218 on a screen of the device 206 represent. The device 206 can also use a speaker 220 configured to play audio associated with the video clip. In this example, the block corresponds to 216 same point in time as the block 210 to which the user 208 may not have spoken yet. Hence the avatar can 218 be shown with his mouth open; however, is from the speaker 220 no audio shown yet. In block 222 that the block 212 corresponds to where the user 208 “Hello” can be said by the avatar 218 are represented in the avatar process with an avatar-specific voice (as described above).

In block 224 can the avatar process those in the block 214 replace identified silence with an avatar-specific word. In this example, the sound of a dog barking (e.g. a recorded or simulated dog barking) may be inserted into the audio (e.g. in place of the silence) so that if it is played while the video clip is being displayed, it will be heard from the speaker 220 a "woof" is represented. In some examples, different avatar-specific words are displayed at 224 based at least in part on the selection of different avatars, and in other examples the same avatar-specific word may be displayed regardless of the selection of the avatar. For example, if the user 208 has kept his mouth open, a "woof" could be displayed if the dog avatar is selected, a "meow" sound could be displayed for a cat avatar, etc. In some cases each avatar can have a predefined sound, that is to play when it is detected that the user 208 has held his or her mouth open for a period of time (e.g. half a second, a whole second, etc.) without speaking. However, in some examples, the process could ignore the open mouth detection for avatars that do not have a predefined effect on that facial expression. In addition, there can be a second stage audio feature analysis that even after the extraction is performed 214 is carried out. For example, if the process determines that a "woof" is to be inserted for a dog avatar (e.g. based on an open mouth detection), the process may also determine how many "woof" sounds to insert (e.g. if the user has kept their mouth open for twice the length of time used to indicate a bark), or if it is not possible to insert the number of bells requested (e.g. in the Scenario of 1 , where the user speaks "bark" to indicate that a "woof" sound should be inserted. Thus, based on the above two examples, it should be apparent that the user 208 Can control effects of the playback (e.g. the recorded avatar message) with his facial expression and his vocal expression. Furthermore, although shown in FIG 1 or 2 not expressly shown, with software for performing the avatar process (e.g. recording the A / V information, extracting features, analyzing the data, implementing the logic, revising the audio and / or video files, and displaying the previews), as well as with Software may be configured to run an application (e.g., an avatar application with its own UI) which enables the user to form the avatar messages and then send them to other user devices.

3 is a simplified block diagram 300 which illustrates components (e.g., software modules) used in the avatar process described above and below. In some examples, more or fewer modules can be used to implement the provision of audio and / or video effects based at least in part on audio and / or video characteristics captured in a recording of a user. In some examples, the device may 302 with a camera 304 , a microphone 306 and a display screen configured to present a UI and the avatar previews (e.g., the initial preview before recording and the preview of the recording before broadcast). In some examples, the avatar process is with an avatar machine 308 and a voice machine 310 configured. The Avatar Machine 308 can manage the list of avatars that process video features (e.g., the facial feature characteristics), revise video information, with the voice machine 301 communicate when indicated and a video of the avatar 312 represent when all processing has been completed and the effects have been implemented (or deprecated). Revising the video information can include adjusting or otherwise editing the metadata associated with the video file. In this way, when the video metadata is used (adjusted or not) to represent the doll, the facial features can be implemented on the doll. In some examples, the voice machine can 310 store the audio information, execute the logic to determine which effects to implement, revise the audio information, and modified audio 314 deploy when all processing is complete and effects have been implemented (or deprecated).

In some examples, once the user selects to record a new avatar video clip, the video features 316 through the camera 304 and the audio characteristics 318 can through the microphone 306 be included. In some cases, there may be fifty (or more) facial features that are within the video features 316 are to be recorded. Example video features include, but are not limited to, duration of expressions, mouth open, frowns, smiles, raised or furrowed eyebrows, and so on. In addition, the video features close 316 possibly only metadata identifying each of the facial features (e.g. data points indicating which places on the user's face have moved or are where and in what position). Furthermore, the video features 316 to the avatar machine 308 and the voice machine 310 to get redirected. At the avatar machine 308 can change the metadata related to the video characteristics 316 belong, be stored and analyzed. In some examples, the avatar engine 308 perform feature extraction from the video file prior to saving the metadata. In other examples, however, feature extraction can be performed before the video features 316 to the avatar machine (in which case the video characteristics 316 the metadata itself). On the voice machine 310 can use the video features 316 with the audio features 318 if it is helpful to match which audio features correspond to which video features (e.g. to see whether certain audio and video features occur simultaneously).

In some cases, audio features also become the voice machine for storage 310 forwarded. Example audio features include, but are not limited to, strength, pitch, dynamic (e.g., changes in power, pitch, voicing, formants, duration, etc.). A raw audio 320 includes the unprocessed audio file as it is recorded. The raw audio 320 can for further processing and for possible (e.g. final) revision to the voice machine 310 and it can also be saved separately so that the original audio can be used if desired. The raw audio 320 can also become a voice recognition module 322 to get redirected. The voice recognition module 322 can be used to recognize words and identify a user's intention from their voice. For example, the voice recognition module 322 determine when a user is angry, sad, happy, or the like. In addition, the voice recognition module 322 when a user utters a keyword (e.g., "bark" as described above), capture it. Information obtained by the voice recognition module 322 captured and / or collected can then be sent to the voice machine for further logic and / or further processing 310 to get redirected. As noted, in some examples, even audio features are extracted in real time during the preview. These audio features can be avatar-specific and can only be generated when a preview of the associated avatar is viewed. The audio features can be avatar-independent and can be generated for all avatars. The audio signal can also be adjusted based in part on these real-time audio feature extractions and with the pre-stored extracted video features generated during or after the recording process but prior to viewing the preview. In addition, the extraction of some features during the Representation at 336 through the voice machine 310 be performed. Some sounds stored in advance 338 can from the voice machine 310 used, if appropriate, to fill in the gaps or to replace other sounds that have been extracted.

In some examples, the voice machine leads 310 determining what to do with the information received from the voice recognition module 322 were extracted. In some examples, the voice machine can 310 the information from the voice recognition module 322 to a feature module 324 forward to determine which features correspond to the data received by the voice recognition module 322 extracted. For example, the feature module 324 indicate (e.g. based on a set of rules and / or logic) that a sad voice generated by the voice recognition module 322 corresponds to raising the pitch of the voice or slowing down the speed or beat of the voice. In other words, the feature module 322 convert the extracted audio features into special voice features. Then an effect type module 326 convert the special voice characteristics to the desired effect. The voice machine 310 may also be responsible for saving each special voice for each possible avatar. For example, there may be standard or hardware coded voices for each avatar. When a user selects a particular avatar, the vote machine can 310 select the appropriate default voice to use during playback without making any other changes. In this case, the modified audio can 314 just the raw audio 320 which is converted to the appropriate avatar voice based on the selected avatar. As the user scrolls through the avatars and selects different ones, the voice machine can 310 the raw audio 320 Modify it spontaneously so that it sounds like the newly selected avatar. Thus, the voice machine must 310 the avatar type 328 provided to make this change. However, if there is an effect to be provided (e.g. changing the pitch, tone, or actual words within the audio file), the voice machine can 310 the raw audio file 320 revise and the modified audio 314 provide. In some examples, On / Off 330 provided an option to use the original audio file. If the user selects “Off” (e.g. effects off), then the raw audio 320 with a video of the avatar 312 (which e.g. corresponds to the unchanged video) can be combined to form the A / V output 332 to create. The A / V output 332 can be deployed to the avatar application on the device's UI 302 is pictured.

The Avatar Machine 308 may be responsible for providing the initial avatar image based at least in part on the avatar type selection 328 . Additionally is the avatar machine 308 responsible for the video characteristics 316 to implement on the matching facial markings of each avatar. For example, if the video characteristics 316 indicate that the user is smiling, the metadata indicating a smile can be applied to the mouth area of the selected avatar so that the avatar is included in the video of the avatar 312 seems to smile. In addition, the avatar machine 308 Timing changes 334 received by the voice machine when appropriate. If, for example, the voice machine 310 determines to apply a voice effect that makes the audio more like a whispering voice (e.g. based on the feature module 324 and / or the effect type 326 and / or the avatar type), and modifying the voice to sound more like a whispered voice, this effect change may include slowing down the voice itself, in addition to decreased strength and other formant and pitch changes. Accordingly, the voice engine can generate modified audio that is slower in playback speed compared to the original audio file for the audio clip. In this scenario you would need the voice machine 310 the avatar machine 308 about timing changes 334 instruct so that the video file can be slowed down appropriately; otherwise the video and audio would be out of sync.

As indicated, a user can use the device's avatar application 302 use to select different avatars. In some examples, the voice effect may change based at least in part on this selection. However, in other examples, the user may be given the opportunity to select a different voice for a given avatar (e.g., the cat voice for the dog avatar, etc.). This type of free form voice effect change can be performed by the user via a selection on the UI or in some cases voice activation or facial movement. For example, a certain facial expression could trigger the voice machine 310 change the voice effect for a given avatar. Furthermore, the voice machine 310 in some examples be configured to make children's voices sound higher or, alternatively, determine that a child's voice should not sound higher because it would sound inappropriately given the fact that the raw audio 320 could already sound high to a child's voice. Making this custom determination of an effect could be driven in part by the extracted audio features, and in that case such features could Include pitch values and ranges across the record.

In some examples, the voice recognition module 322 include a recognition engine, a word recognizer, a voice pitch analyzer, and / or a formant analyzer. By the voice recognition module 322 Analysis performed can identify whether the user is excited, angry, happy, etc. In addition, the voice recognition module 322 be able to identify context and / or intonation of the user's voice, as well as change the intention of a choice of words and / or determine a profile (e.g. a virtual identity) of the user.

In some examples, the avatar process 300 be configured so that the video clip is bundled / displayed by the video of the avatar 312 and either the modified audio 314 or the raw audio 320 in the A / V output 332 be combined. In order to join the two, the voice machine has to be used 310 just know an identifier for the metadata associated with the avatar's video 312 belong (e.g. it doesn't necessarily need the avatar's video 312 , it only needs the identification of the metadata). A message within a messaging application (e.g. the avatar application) can be sent to other computing devices where the message is A / V output 332 includes. When a user selects a “Send” option in the UI, the last video clip to be previewed can be sent. For example, if a user previews their video clip with the dog avatar and then switches to the cat avatar preview, the cat avatar video will be sent when the user selects "Send". In addition, the status of the last preview can be saved and used later. For example, if a special effect was used on the last message sent (e.g. an avatar video clip), the first preview of the next message that is generated can use that special effect.

The one from the voice machine 310 and / or the avatar machine 308 implemented logic can check for certain cues and / or features and then revise the audio and / or video files to implement the desired effect. Some example feature / effect pairs include: Detecting that the user has opened their mouth and pauses for a moment. In this example, both facial feature characteristics (e.g. mouth opened) and audio feature characteristics (e.g. silence) must occur simultaneously in order for the desired effect to be implemented. For this feature / effect pair, the desired effect is to rework the audio and video so that the avatar appears to be making an avatar / animal-specific sound. For example, a dog makes a bell sound, a cat makes a meow sound, a monkey, horse, unicorn, etc. make the appropriate sound for this character / animal. Other example feature / effect pairs include lowering the audio pitch and / or tone when a frown is detected. In this example, only the video feature characteristics need to be acquired. In some examples, however, this effect could be implemented at least in part based on the voice recognition module 322 Sadness captured in the user's voice. In this case, there would be no video features at all 316 needed. Other example feature / effect pairs include Whisper, which causes the audio and video speeds to slow down, attenuate, and / or cause the changes to decrease. In some cases video changes can lead to modifications of the audio, while in other cases audio changes can lead to modifications to the video.

As stated above, the avatar machine 308 to act as the feature extractor in some examples, in which case prior to sending to the avatar engine 308 no video features 316 and audio features 318 must exist. Instead, the raw audio 320 and metadata associated with the raw video into the avatar engine 308 being forwarded, being the avatar machine 308 extract the audio feature characteristics and the video feature characteristics (e.g., facial feature characteristics). In other words, although in 3 not drawn this way, parts of the avatar machine 308 actually inside the camera 304 to be available. Additionally, in some examples, metadata pertaining to the video characteristics 316 belong to be stored in a secure container, and if the voice machine 310 is running, it can read the metadata from the container.

In some cases, because the avatar's preview video clip is not displayed in real time (e.g., it is rendered and displayed after the video is recorded, and sometimes only in response to a play option being selected), the audio and video Video information is processed offline (e.g. not in real time). Hence, the avatar machine 308 and the voice machine 310 Read the audio and video information beforehand and make contextual decisions in advance. Then the voice machine can 310 revise the audio file accordingly. This ability to read ahead of time and make decisions offline significantly increases the efficiency of the system, especially for lengthy recordings. Additionally this enables a second stage of analysis, whereby additional logic can be processed. Thus, the entire audio file can be analyzed before any final decisions are made. For example, if the user says "bark" twice in a row but the words "bark" are spoken too closely together, the actual "woof" sound that was pre-recorded may not be able to fit in with the time that the user needed to say "bark, bark". In this case the voice machine can 310 the voice recognition information 322 and determine to ignore the second "bark", as it is not possible to incorporate both "woof" sounds into the audio file.

As stated above, when the audio file and video are bundled, the voice engine needs to be the A / V output 332 not necessarily on the avatar's video 312 access. Instead, when the video is played, the video file (e.g., a .mov format file or the like) is created by accessing an array of features (e.g., floating point values) written to the metadata file. However, any permutations / adjustments to the audio and video files can be made in advance, and some can even be made in real time as the audio and video are extracted. Additionally, in some examples, each modified video clip could be temporarily stored (cached) so that if the user re-selects an avatar that they have already previewed, the processing to create / display that particular preview is not repeated got to. As opposed to redisplaying the revised video clip every time the same avatar is selected during the preview section, the above-mentioned caching of displayed video clips would enable great savings in processor power and instructions per second (IPS) to be realized, especially for longer recordings and / or records with a large number of effects.

Additionally, in some examples, noise suppression algorithms can be used to handle cases where the microphone is used 306 recorded sounds include sounds other than the user's voice. For example, if the user is in a windy area or in a noisy room (e.g. a restaurant or bar). In these examples, a noise reduction algorithm could reduce the decibels output of certain parts of the audio recording. Alternatively or additionally, different voices could be separated and / or only audio could be recorded which comes from certain viewing angles (e.g. from the corner of the user's face) and other voices could be ignored or suppressed. In other cases, if in the avatar process 300 If the noise levels are determined to be too loud or will be difficult to process, the recording option is through the process 300 deactivated.

4th illustrates a sample flow chart that represents a process 400 Figure 12 for implementing various audio and / or video effects based at least in part on audio and / or video features, according to at least a few examples. In some examples, the computing device may 106 the 1 or some other similar user device (e.g., which can at least run the avatar process 300 the 3 used) the process 400 the 4th carry out.

In block 402 can the computing device 106 record a video that has an audio component. In some examples, the video and audio can be picked up by two different hardware components (e.g., a camera can pick up the video information while a microphone can pick up the audio information). In some cases, however, a single piece of hardware can be configured to record both audio and video. In any case, the video and audio information can be associated with one another (for example by means of a common identifier, a time stamp or the like). Thus, the video can have an audio component (e.g. they are part of the same file), or the video can be associated with an audio component (e.g. two files that go together).

In block 404 can the computing device 106 Extract facial features and audio features from recorded video and audio information, respectively. In some cases, the facial feature information can be obtained through the avatar machine 308 extracted and saved as metadata. The metadata can be used to implement any facial feature on a special doll or on any animation or virtual face. Thus, the actual video file need not be saved, creating storage efficiency and significant savings. With regard to audio feature extraction, a voice recognition algorithm can be used to extract various voice features; for example words, expressions, pitch, speed, etc.

In block 406 can the computing device 106 Capture context from the extracted features. For example, the context can be an intention of a Include user, mood, situation, place, background elements, ideas, etc. Context can be important when using logic to determine which effects to apply. In some cases, the context can be combined with captured spoken words to determine whether and / or how to adjust the audio file and / or the video file. In one example, a user may frown and speak slowly. The frowning of the eyebrows is a video feature that is featured in the block 404 may have been extracted, and the slow speech is an audio feature that is in the block 404 may have been extracted. Individually, these two features could mean something different; however, if they are combined with each other, it can be determined by the avatar process that the user is concerned about something. In this case, the context of the message could be that a parent is talking to a child or a friend is talking to another friend about a serious or worrisome topic.

In block 408 can the computing device 106 Determine effects for displaying the audio and / or video files based at least in part on the context. As indicated above, one context could be of concern. A special video and / or audio feature can thus be used for this effect. For example, the voice file can be adjusted to make the voice sound darker or to slow it down. In other examples, the avatar-specific voice could be replaced with a version of the original audio (e.g., raw audio) to convey the seriousness of the message. Various other effects can be used for various other contexts. In other examples, the context may be animal sounds (e.g., based on the user speaking “barking” or “meowing” or the like). In this case the particular effect would be to replace the spoken word “bark” with the sound of a dog barking.

In block 410 can the computing device 106 perform additional logic for additional effects. For example, if the user attempted to trigger the barking effect by saying "bark" twice in a row, the additional logic may need to be used to determine whether the additional bark is technically feasible. As an example, if the audio clip of barking used to replace the spoken word in the raw audio information is 0.5 seconds long but the user speaks "bark" twice in a 0.7 second period, the additional Logic determine that no two bells can fit in the available 0.7 seconds. It may therefore be that the audio and video files have to be expanded to accommodate both bells, that the bells must be shortened (e.g. by processing the stored bells) or that the second spoken word "bark" must be ignored .

In block 412 can the computing device 106 revise the audio and / or video information based at least in part on the determined effects and / or additional effects. In some examples, only one set of effects may be used. In either case, however, the raw audio file can be adapted (e.g. revised) to form a new audio file in which additional sounds have been added and / or sounds have been removed. For example, in the “barking” application, the spoken word “barking” is removed from the audio file and a new sound is inserted that represents an actual dog barking. The new file can be saved with a different identifier or with an identifier appended (for example, the raw audio identifier with a .v2 identifier to indicate that it is not the original). In addition, the raw audio file is saved separately so that it can be reused for additional avatars and / or if the user chooses not to use the particular effects.

In block 414 can the computing device 106 receive a selection of an avatar from the user. The user can access the avatar application through a UI run by the computing device 106 select one of a variety of different avatars. The avatars can be selected using a scroll wheel, a drop-down menu or an icon menu (e.g. where each avatar is visible in its own position on the screen).

In block 416 can the computing device 106 display the revised video with the revised audio based at least in part on the selected avatar. In this example, each customized video clip (e.g., a finished video clip for the avatar that has customized audio and / or video) can be generated for each corresponding avatar prior to the user's selection of the avatar. In this way, the processing has already been completed and the adapted video clip is ready to be displayed immediately after the avatar has been selected. While this might require additional IPS prior to avatar selection, it speeds up rendering. Additionally, processing of each customized video clip can be performed while the user is viewing the first preview (e.g. the preview corresponding to the first avatar / default avatar presented in the UI).

5 illustrates a sample flow chart that represents a process 500 Figure 12 for implementing various audio and / or video effects based at least in part on audio and / or video features, according to at least a few examples. In some examples, the computing device may 106 the 1 or some other similar user device (e.g., which can at least run the avatar process 300 the 3 used) the process 500 the 5 carry out.

In block 502 can the computing device 106 record a video that has an audio component. Just like in the block 402 the 4th For example, the video and audio can be recorded by two different hardware components (e.g. a camera can record the video information while a microphone can record the audio information). As indicated, the video can have an audio component (e.g., they are part of the same file), or the video can be associated with an audio component (e.g., two files that go together).

In block 504 can the computing device 106 Extract facial features and audio features from recorded video and audio information, respectively. Just as above, the facial feature information can be made via the avatar machine 308 extracted and saved as metadata. The metadata can be used to implement any facial feature on a special doll or on any animation or virtual face. Thus, the actual video file need not be saved, creating storage efficiency and significant savings. With regard to audio feature extraction, a voice recognition algorithm can be used to extract various voice features; for example, words, phrases, pitch, speed, etc. Additionally, in some examples, the avatar machine 308 and / or the voice machine 310 perform the audio feature extraction.

In block 506 can the computing device 106 Capture context from the extracted features. For example, the context can include a user's intention, mood, situation, location, ideas, identity, and so on. Context can be important when using logic to determine which effects to apply. In some cases, the context can be combined with spoken words to determine whether and / or how to adjust the audio file and / or the video file. In one example, the age of a user can be captured as the context (e.g. child, adult, etc.) based at least in part on facial and / or voice characteristics. For example, a child's face may have specific features that can be identified (e.g., large eyes, a small nose and a relatively small head, etc.). In this way, a children's context can be captured.

In block 508 can the computing device 106 receive a selection of an avatar from the user. The user can access the avatar application through a UI run by the computing device 106 select one of a variety of different avatars. The avatars can be selected using a scroll wheel, a drop-down menu or an icon menu (e.g. where each avatar is visible in its own position on the screen).

In block 510 can the computing device 106 Determine effects for displaying the audio and / or video files based at least in part on the context and the selected avatar. In this example, the effects for each avatar can be created after each avatar is selected, rather than all at once. In some cases, this enables significant processor and memory savings to be realized since only one group of effects is performed at a time and only one avatar is displayed at a time. These savings can be realized in particular if the user does not select multiple avatars for preview.

In block 512 can the computing device 106 perform additional logic for additional effects, similar to the above with respect to the block 410 the 4th described. In block 514 can the computing device 106 revise the audio and / or video information based at least in part on the determined effects and / or additional effects for the selected avatar, similar to the above with respect to the block 412 the 4th described. In block 516 can the computing device 106 display the revised video with the revised audio based at least in part on the selected avatar, similar to the above with respect to the block 416 the 4th described.

In some examples it can go through the avatar process 300 determine whether the expiration 400 or the process 500 to be performed based at least in part on historical information. For example, if the user generally uses the same avatar every time, the flow is 500 more efficient. However, if the user is regularly switching between avatars and previewing several different avatars per video clip, then the process can be more efficient 400 to follow.

6th illustrates a sample UI 600 to allow a user to use the To use avatar application (e.g. according to the avatar application possibility 602 ). In some examples, the UI 600 look different (e.g. it may appear as a standard text (e.g. short messaging service (SMS)) messaging application) until the avatar application facility 602 is selected. As indicated, the avatar application can interact with the avatar process (e.g. the avatar processn 300 the 3 ) communicate to make requests for recording, processing (e.g. extracting features, running logic, etc.) and adjusting audio and / video. For example, when the user selects a recording option (e.g. a video clip record / send option 604 ), the avatar application can make a call through an application programming interface (API) to the avatar process to begin capturing video and audio information using the appropriate hardware components. In some examples, the video clip recording / sending capability 604 before the start of the recording session as a red circle (or a simple circle without the in 6th line shown). That way, the facility looks more like a standard record button. While the session is being recorded, the appearance of the video clip record / send facility may change 604 can be changed to look like a countdown on a clock or some other representation of a timer (e.g. if the length of the video clip recordings is limited). In other examples, however, the video clip record / send facility 604 just change colors to indicate that the avatar application is recording. If there is no timer or limit on the length of the recording, the user may have the video clip record / send option 604 must select again to stop recording.

In some examples, a user can choose the avatar option 606 use to select an avatar. This can be done before the avatar video clip is recorded and / or after the avatar video clip has been recorded. If the selection is made prior to recording, the initial preview of the user's movements and facial characteristics is presented as the selected avatar. In addition, the recording is performed while a live preview (e.g., a real-time preview) of the recording is displayed, with the user's face being represented by the selected avatar. Once the recording is complete, a second preview (e.g. a playback of the current recording) is displayed, again using the selected avatar. At this stage, however, the user can use the avatar selection option 606 Scroll to select a new avatar to view the recording preview. In some cases, after selecting a new avatar, the UI will begin previewing the recording using the selected avatar. The new preview can be displayed with the audio-video effects or as originally recorded. As indicated, the determination of whether to display the effect version or the original can be based at least in part on the last playback method used. For example, if effects were used the last time they were played, effects can be used the first time they were played after a new avatar was selected. However, if no effects were used the last time it was played, the first time it was played after selecting a new avatar, no effects may be used. In some examples, the user can play the video clip with effects by having an effect preview option 608 selects, or without effects, by having an original preview option 610 selects. Once satisfied with the video clip (e.g. the message), the user can send the avatar video in a message to another computing device by using the video clip record / send option 604 used. The video clip is sent using the format that corresponds to the last preview (for example, with or without effects). A video clip deletion option is available at any time if the user so desires 612 can be selected to delete the avatar video and either start or exit the avatar and / or messaging applications from.

7th FIG. 10 illustrates an example flow diagram showing a process (e.g., a computer implemented method) 700 for implementing various audio and / or video effects based at least in part on audio and / or video features, according to at least a few examples. In some examples, the computing device may 106 the 1 or some other similar user device (e.g. using at least one avatar application similar to that in 6th shown and the avatar process 300 the 3 ) the process 700 the 7th To run.

In block 702 can the computing device 106 display an interface for creating virtual avatars. The interface for creating virtual avatars can look similar to the one in 6th illustrated UI. However, any UI configured to enable the same features described herein can be used.

In block 704 can the computing device 106 display a first preview content of a virtual avatar. In some examples, the first preview content may be a real-time representation of the Face of the user, including movement and facial expressions. However, the first preview would provide an avatar (e.g. cartoon character, digital / virtual doll) to represent the user's face rather than an image of the user's face. The first preview can only be a video or at least a representation of the avatar without sound. In some examples, this initial preview is not recorded and can be used for as long as the user desires without limitation other than battery life or computing device storage space 106 .

In block 706 can the computing device 106 the selection of an input (e.g. the video clip recording / sending option 604 the 6th ) in the interface for creating virtual avatars. This selection can be made while the UI is displaying the first preview content.

In block 708 can the computing device 106 begin to record video and audio signals based at least in part on the input provided in the block 706 is captured. As described, the video and audio signals can be recorded by suitable hardware components and can be recorded by one or a combination of such components.

In block 710 can the computing device 106 Extract audio feature characteristics and facial feature characteristics as detailed above. As indicated, the extraction can be done by special modules of the avatar process 300 the 3 or by other extraction and / or analysis components of the avatar application and / or the computing device 106 be performed.

In block 712 can the computing device 106 generate adjusted audio signals based at least in part on facial feature characteristics and audio feature characteristics. For example, the one in the block 708 recorded audio file can be permanently (or temporarily) revised (e.g. adjusted) to include new sounds, new words, etc., and / or that the original pitch, original sound, original volume, etc. are adjusted. These adjustments can be made based at least in part on the context, which is captured via an analysis of the facial feature characteristics and the audio feature characteristics. Additionally, the adjustments may be based on the selected avatar type and / or based on specific movements, facial expressions, words, phrases, or actions performed by the user (e.g., expressed through the user's face) during the recording session.

In block 714 can the computing device 106 generate a second preview content of the virtual avatar in the UI according to the customized audio signal. The generated second preview content can be based at least partially on the currently selected avatar or any standard avatar. As soon as the second preview content is generated, the computing device can 106 in block 716 display the second preview content in the UI.

8th Fig. 10 illustrates an example flowchart depicting a process (e.g., instructions stored on computer readable memory and executable) 800 for implementing various audio and / or video effects based at least in part on audio and / or video features, according to at least a few examples. In some examples, the computing device may 106 the 1 or some other similar user device (e.g. using at least one avatar application similar to that in 6th and the avatar process 300 the 3 ) the process 800 the 8th carry out.

In block 802 can the computing device 106 detect a request to create an avatar video clip of a virtual avatar. In some examples, the request may be based, at least in part, on selecting a video clip record / broadcast option 604 the 6th based on the user.

In block 804 can the computing device 106 record a video signal associated with a face in the field of view of the camera. In block 806 can the computing device 106 record an audio signal that corresponds to the video signal (e.g. coming from the face recorded by the camera).

In block 808 can the computing device 106 Extract voice feature characteristics from the audio signal, and in block 810 can the computing device 106 Extract facial feature characteristics from the video signal.

In block 812 can the computing device 106 capture a request to preview the avatar video clip. This requirement can at least in part be based on the selection of a new avatar by a user via the avatar selection option 606 the 6th are based or at least partly on the selection of the effect preview option 608 the 6th based on a user.

In block 814 can the computing device 106 generate adjusted audio signals based at least in part on facial feature characteristics and voice feature characteristics. For example, the one in the block 806 The recorded audio file can be revised (e.g. adjusted) to include new sounds, new words, etc., and / or so that the original pitch, original sound, original volume, etc. are adjusted. These adjustments can be made based at least in part on the context captured via an analysis of the facial feature characteristics and the voice feature characteristics. Additionally, the adjustments may be based on the selected avatar type and / or based on specific movements, facial expressions, words, phrases, or actions performed by the user (e.g., expressed through the user's face) during the recording session.

In block 816 can the computing device 106 preview the virtual avatar in the UI according to the customized audio signal. The preview generated can be based at least in part on the currently selected avatar or any standard avatar. As soon as the preview is generated, the computing device can 106 in the block 816 also display the second preview content in the UI.

9 is a simplified block diagram showing an example architecture 900 for implementing the features described herein in accordance with at least one embodiment. In some examples, the computing device may 902 (e.g. the computing device 106 the 1 ) showing the example architecture 900 be configured to display relevant UIs, ingest audio and video information, extract relevant data, perform logic, revise the audio and video information, and display Animoji videos.

The computing device 902 may be configured to run or otherwise manage applications or instructions for performing the techniques described, such as: B., but not limited to, providing a user interface (e.g. the user interface 600 the 6th ) to record, preview and / or send video clips of virtual avatars. The computing device 602 may receive input from a user at the user interface (e.g., using one or more I / O devices 904, such as a touch screen), record information, process the information, and then preview the video clips, too using the I / O device (s) 904 (e.g., a speaker of the computing device 902 ). The computing device 902 may be configured to revise audio and / or video files based at least in part on facial features extracted from the captured video and / or voice features extracted from the captured audio.

The computing device 902 can be any type of computing device such as E.g., but not limited to a mobile phone (e.g. a smartphone), a tablet computer, a personal digital assistant (PDA), a laptop computer, a desktop computer, a thin client device, a smart watch, wireless headset, or the like.

In an illustrative configuration, the computing device 902 at least one memory 914 and one or more processing units (or one or more processors) 916. The processor (s) 916 can be implemented in hardware, computer-executable instructions, or combinations thereof as appropriate. Computer executable instruction or firmware implementations of processor (s) 916 may include computer executable or machine executable instructions written in any suitable programming language for performing the various functions described.

The memory 914 may store program instructions that are loadable into and executable on processor (s) 916, as well as data generated during the execution of those programs. Depending on the configuration and type of computing device 902 can the memory 914 volatile (such as RAM random access memory) and / or non-volatile (such as read only memory (ROM), flash memory, etc.). The computing device 902 can also have additional removable storage and / or non-removable storage 926 include, including but not limited to, magnetic storage, optical disks, and / or tape storage. The disk drives and their associated non-transitory computer readable media can provide non-transitory storage of computer readable instructions, data structures, program modules, and other data for the computing devices. In some implementations, the memory 914 include several different types of storage, such as Static random access memory (SRAM), dynamic random access memory (DRAM), or ROM. Although the volatile memory described herein may be referred to as RAM, any volatile memory that would no longer store data stored therein once disconnected from a host and / or power supply would be suitable.

The memory 914 and the additional storage 926 , both removable and non-removable, are examples of permanent computer readable storage media, respectively. For example, permanent computer-readable storage media can include volatile or non-volatile, removable or non-removable media that can be used in any method or technology for storing information, such as. B. computer-readable instructions, data structures, program modules or other data can be implemented. The memory 914 and the additional storage 926 are both examples of permanent computer readable storage media. Additional types of computer storage media used in the computing device 902 phase change RAM (PRAM), SRAM, DRAM, RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or another storage technology, a compact Read-only disc storage (CD-ROM), digital video disc (DVD) or other optical storage, magnetic cartridges, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information to be stored, and to which by the computing device 902 can be accessed. Combinations of any of the foregoing should also be included within the scope of persistent computer readable storage media.

Alternatively, computer readable communication media can include computer readable instructions, program modules, or other data sent within a data signal, such as, for example, a. B. a carrier wave or other transmission. However, as used herein, computer readable storage media does not include computer readable communication media.

The computing device 902 can also be a communication link (communication links) 928 included, which enables the computing device 902 communicates over one or more networks with a data store, other computing device or server, user terminal devices, and / or other devices. Such networks can be any or a combination of many different types of networks, such as Cable networks, the Internet, wireless networks, cellular networks, satellite networks, other private and / or public networks, or any combination thereof. The computing device 902 may also include an I / O device (s) 904, such as A touch input device, a keyboard, a mouse, a pen, a voice input device, a display, a speaker, a printer, etc.

If you look at the contents of the store 914 Considered in more detail, the memory 914 an operating system 932 and / or include one or more application programs or services for implementing the features disclosed herein, including a user interface module 934 , an avatar control module 936 , an avatar application module 938 and a message module 940 . The memory 914 can also be configured to store one or more audio and video files to be used to produce audio and video output. In this way, the computing device 902 perform all of the operations described herein.

In some examples, the user interface module 934 be configured to use the user interface of the computing device 902 manage. For example, the user interface module 934 represent any number of different UIs used by the computing device 902 be requested. In particular, the user interface module 934 be configured to use the UI 600 the 6th that enables implementation of the features described herein, including communication with the avatar process 300 the 3 which is responsible for capturing video and audio information, extracting appropriate facial and voice characteristic information, and revising the video and audio information prior to displaying the generated avatar video clip as described above.

In some examples, the avatar control module is 936 configured to use the avatar process 300 to implement (e.g. carry out instructions for its implementation) while the Avatar application module 938 configured to implement the user face application. As indicated above, the Avatar application module 938 one or more APIs for requesting and / or providing information for the avatar control module 936 to use.

In some embodiments, the messaging module 940 implement any standalone or add-on messaging application using the Avatar Control Module 936 and / or the avatar application module 938 can communicate. In some examples, the messaging module 940 complete with the Avatar application module 938 be integrated (such as in the UI 600 the 6th it appears that the Avatar application is part of the messaging application. In other examples, however, the news application may 940 the avatar Application module 938 invoke when a user requests to create an avatar video clip and the avatar application module 938 can open a completely new application that uses the messaging module 940 is integrated.

The computing device 902 can also be equipped with a camera and microphone, as at least in 3 shown and the processors 916 may be configured to execute instructions to display an initial preview of a virtual avatar. In some examples, while the first preview of a virtual avatar is displayed, input may be captured through a virtual avatar creation interface provided by the user interface module 934 is pictured. In some cases the avatar control module 936 in response to detecting the input to the virtual avatar generation interface, starting a recording session including: recording a video signal associated with a face in a field of view of the camera via the camera, recording an audio signal via the microphone which corresponds to the recorded video signal extracting audio feature characteristics from the recorded audio signal and extracting facial feature characteristics associated with the face from the recorded video signal. In addition, the avatar control module 936 in response to detecting a course of the recording session, generate an adjusted audio signal based at least in part on the audio feature characteristics and the facial feature characteristics, and according to the facial feature characteristics and the adjusted audio signal, a second preview of the virtual avatar in the interface for generation of virtual avatars.

Illustrative methods, computer readable medium, and systems for providing various techniques for customizing audio and / or video content based at least in part on voice and / or facial feature characteristics are described above. Some or all of these systems, media and processes can, but do not have to, at least in part through architectures and processes such as B. those implemented at least in 1 to 9 are shown. While many of the embodiments are described above in relation to news applications, it should be understood that any of the above techniques can be used within any type of application, including real-time video playback applications or real-time video news applications. For purposes of explanation, numerous specific configurations and details are set forth in order to provide a thorough understanding of the examples. However, it should be apparent to those skilled in the art that the examples can be carried out without the specific details. Furthermore, well-known features have sometimes been omitted or simplified in order not to obscure the example being described.

The various embodiments can also be implemented in a wide variety of operating environments, which in some cases may include one or more user computers, computing devices, or processing devices that can be used to run any of a number of applications. User or client devices can be any of a number of general purpose personal computers, such as. Include desktop or laptop computers running a standard operating system as well as cellular, wireless and handheld devices running mobile software and capable of supporting a number of network and messaging protocols. Such a system may also include a number of workstations on which any of a variety of commercially available operating systems and other known applications for such purposes as B. Development and database administration in progress. These devices can also include other electronic devices, such as e.g. E.g., dummy end devices, thin clients, gaming systems, and other devices capable of communicating over a network.

In most embodiments, at least one network known to those skilled in the art will be used to support communications using any of a variety of commercially available protocols, e.g. B. TCP / IP, OSI, FTP, UPnP, NFS, CIFS and AppleTalk. The network can be, for example, a local area network, a wide area network, a virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, an infrared network, a wireless network, and any combination thereof.

In embodiments using a network server, any of a variety of server or mid-tier applications can run on the network server, including HTTP servers, FTP servers, CGI servers, data servers, Java servers and business application servers. The server (s) may also be capable of executing programs or scripts in response to requests from user devices, e.g. By running one or more applications which can be implemented as one or more scripts or programs written in any programming language are, such as B. Java ^® , C, C # or C ++ or any scripting language such as. B. Perl, Python or TCL and combinations thereof. The applicant (s) server (can) also include database servers, including but not limited to, those that are commercially available from Oracle ^®, Microsoft ^®, Sybase ^®, and IBM ^®.

The environment can include a variety of data stores and other storage and storage media, as described above. These can be located in a variety of locations, such as. On a storage medium local to (and / or located in) one or more of the computers or remotely from any or all of the computers on the network. In a specific group of embodiments, the information can be located in a storage area network (SAN) familiar to those skilled in the art. Similarly, any files necessary to perform the functions attributed to the computers, servers, or other network devices may be stored locally and / or remotely, if necessary. Where a system includes computerized devices, each such device may include hardware elements that may be electrically connected by a bus, the elements, for example, at least one central processing unit (CPU), at least one input device (e.g., mouse, keyboard , a controller, a touch screen, or a keypad) and at least one output device (e.g., a display device, a printer, or a speaker). Such a system may also include one or more storage devices, such as e.g. Disk drives, optical storage devices, and solid state storage devices such as e.g. B. RAM or ROM, as well as removable media devices, memory cards, flash cards, etc.

Such devices may also include a computer readable storage media reader, communication device (e.g., modem, network card (wireless or wired), infrared communication device, etc.), and memory, as described above. The reading device for computer readable storage media can be connected to a permanent computer readable storage medium or be configured to receive one which represents remote, local, fixed and / or removable storage devices as well as storage media for temporarily or more permanently containing, storing, sending and retrieving computer readable information . The system and the various devices also typically include a number of software applications, modules, services, or other elements located within at least one memory device, including an operating system and application programs, such as, e.g. B. a client application or a browser. It should be understood that alternative embodiments can vary in numerous ways from those described above. For example, customized hardware could also be used and / or certain elements could be implemented in hardware, software (including portable software such as applets), or both. Furthermore, a connection with other computing devices, such as. B. network input / output devices can be used.

Persistent storage media and computer readable storage media for containing code or sections of code may include any suitable media known or used in the art (other than non-permanent media such as carrier waves or the like), such as, e.g. B., but not limited to, volatile and non-volatile, removable and non-removable media used in any method or technology for storing information, such as. Computer readable instructions, data structures, program modules or other data, including RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other storage technology, CD-ROM, DVD or other optical storage , Magnetic cartridges, magnetic tape, magnetic disk storage or magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a system device. Based on the disclosure and teachings provided herein, those of ordinary skill in the art will recognize other ways and / or methods of implementing the various embodiments. However, as indicated above, computer readable storage media do not include non-permanent media, such as. B. carrier waves or the like.

Accordingly, the specification and drawings are to be viewed in an illustrative rather than a restrictive sense. It is apparent, however, that various modifications and changes can be made therein without departing from the broader spirit and scope of the disclosure as set forth in the claims.

Other variations are within the spirit of the present disclosure. Thus, although the techniques disclosed are different Modifications and alternative constructions are amenable to certain illustrated embodiments thereof shown in the drawings and described in detail above. It should be understood, however, that the disclosure is not intended to be limited to the particular form or forms disclosed, but on the contrary is intended to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the disclosure as they come are defined in the accompanying claims.

The use of the terms “a”, “an” and “the”, “the”, “the” and similar reference words in the context of describing the disclosed embodiments (particularly in the context of the following claims) are to be construed as encompassing both the Covers the singular as well as the plural, unless otherwise stated herein and unless the context clearly speaks against it. The terms “comprising,” “having,” “including,” and “containing” are to be construed as open-ended terms (i.e., meaning “including, but not limited to”) unless otherwise specified. The term “connected” is to be interpreted as “partly or wholly contained within ...”, “attached to” or “connected to one another”, even if something is placed in between. The term "based on" should be construed as being candid and in no way limiting and, where appropriate, should be interpreted or otherwise read as "based at least in part on". The provision of ranges of values herein is intended only as a shorthand way of listing each individual value that falls within the range, unless otherwise specified herein, and each individual value is included in the specification as if it were individually listed herein. All of the methods described herein can be performed in any suitable order unless otherwise stated herein or unless the context clearly indicates otherwise. The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended to better explain embodiments of the disclosure, and not to limit the scope of the disclosure unless otherwise stated. No language in the specification should be construed as indicating any unclaimed element as essential to the practice of the disclosure.

A disjunctive language such as However, unless specifically stated otherwise, the term "at least one of X, Y, or Z" is to be understood within the context of its general use to represent that an element, term, etc. is either X , Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is not generally intended to imply that certain embodiments require at least one of X, at least one of Y, or at least one of Z to be present. In addition, a conjunctive language such as B. Unless specifically stated otherwise, the term "at least one of X, Y and Z" should also be understood to mean X, Y, Z or any combination thereof, including "X, Y and / or Z" .

Preferred embodiments of this disclosure are described herein, including the best mode for carrying out the disclosure known to the inventors. Variations of those preferred embodiments will become apparent to those of ordinary skill in the art after reading the preceding description. The inventors expect that those skilled in the art will employ such variations as necessary, and the inventors intend that the disclosure be practiced otherwise than as specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of subject matter set forth in the claims appended hereto as permitted by applicable law. Moreover, all combinations of the elements described above in all possible variations thereof are encompassed by the disclosure, unless otherwise stated herein and unless the context clearly indicates otherwise.

All references, including publications, patent applications and patents cited herein, are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and set forth in their entirety herein.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 15908603 [0001]
• US 16/033111 [0001]

Claims (20)

Method comprising: on an electronic device that has at least one camera and one microphone: Displaying an interface for creating virtual avatars; Displaying a first preview content of a virtual avatar in the interface for generating virtual avatars, the first preview content of the virtual avatar real-time preview video frames of a user portrait recording in a field of view of the camera and corresponding changes in appearance of the portrait photograph; Detecting an input to the interface for creating virtual avatars while displaying the first preview content of the virtual avatar; in response to the capture of the input in the interface for generating virtual avatars: Recording a video signal associated with the user portrait recording during a recording session via the camera; Recording a user audio signal through the microphone during the recording session; Extracting audio feature characteristics from the recorded user audio signal; and Extracting facial feature characteristics related to the Face include, from the recorded video signal; and in response to detecting a history of the recording session: Generating an adjusted audio signal from the recorded audio signal based at least in part on the facial feature characteristics and the audio feature characteristics; Generating a second preview content of the virtual avatar in the interface for generating virtual avatars in accordance with the facial feature characteristics and the adjusted audio signal; and Representation of the second preview content in the interface for generating virtual avatars. Procedure according to Claim 1 further comprising storing facial feature metadata associated with the facial feature characteristics extracted from the video signal and storing audio metadata associated with the audio feature characteristics extracted from the audio signal. Procedure according to Claim 2 , further comprising generating customized facial feature metadata from the facial feature metadata based at least in part on the facial feature characteristics and the audio feature characteristics. Procedure according to Claim 3 wherein the second preview of the virtual avatar is further displayed according to the customized face metadata. An electronic device comprising: a camera; a microphone; and one or more processors in communication with the camera and microphone, the one or more processors configured to: Detecting an input to an interface for generating virtual avatars while displaying a first preview of a virtual avatar; in response to capturing input to the virtual avatar generation interface, starting a recording session including: Taking in via the camera a video signal associated with a face in a field of view of the camera; Recording an audio signal associated with the recorded video signal through the microphone; Extracting audio feature characteristics from the recorded audio signal; and Extracting facial feature characteristics related to the Face include, from the recorded video signal; and in response to detecting a course of the recording session: Generating an adjusted audio signal based at least in part on the audio feature characteristics and the facial feature characteristics; and Displaying a second preview of the virtual avatar in the interface for generating virtual avatars according to the facial feature characteristics and the adjusted audio signal. Electronic device according to Claim 5 wherein the audio signal is further adjusted based at least in part on a type of the virtual avatar. Electronic device according to Claim 6 wherein the type of virtual avatar is received based at least in part on a choice of avatar types presented in the virtual avatar generation interface. Electronic device according to Claim 6 wherein the type of virtual avatar includes an animal species, and wherein the customized audio signal is generated based at least in part on a predetermined sound associated with the animal species. Electronic device according to Claim 5 wherein the one or more processors are further configured to determine whether a part of the audio signal corresponds to the face in the field of view. Electronic device according to Claim 9 wherein the one or more processors are further configured to store the portion of the audio signal for use in generating the customized audio signal, in accordance with a determination that the portion of the audio signal corresponds to the face. Electronic device according to Claim 9 wherein the one or more processors are further configured to discard at least the portion of the audio signal in accordance with a determination that the portion of the audio signal does not correspond to the face. Electronic device according to Claim 5 wherein the audio feature characteristics include features of a voice associated with the face in the field of view. Electronic device according to Claim 5 wherein the one or more processors are further configured to store facial feature metadata associated with the facial feature characteristics extracted from the video signal. Electronic device according to Claim 13 wherein the one or more processors are further configured to store customized facial metadata based at least in part on the facial feature characteristics and the audio feature characteristics. Electronic device according to Claim 14 wherein the second preview of the virtual avatar is generated according to the customized face metadata and the customized audio signal. Computer readable storage medium that stores computer executable instructions that, when executed by one or more processors, configure the one or more processors to perform operations, comprising: in response to detecting a request to create an avatar video clip of a virtual avatar: Picking up a video signal associated with a face in a field of view of a camera of an electronic device via the camera; Picking up an audio signal through a microphone of the electronic device; Extracting voice feature characteristics from the recorded audio signal; and Extracting facial feature characteristics related to the face belong, from the recorded video signal; and in response to detecting a request to preview the avatar video clip: Generating an adjusted audio signal based at least in part on the facial feature characteristics and the voice feature characteristics; and Preview the video clip of the virtual avatar using the customized audio. Computer readable storage medium according to Claim 16 wherein the audio signal is adjusted based at least in part on a facial expression identified in the facial feature characteristics associated with the face. Computer readable storage medium according to Claim 16 wherein the adjusted audio signal is further adjusted by inserting one or more pre-stored audio samples. Computer readable storage medium according to Claim 16 wherein the audio signal is adjusted based at least in part on a level, pitch, duration, variable playback speed, speech spectral format positions, speech spectral format levels, instant playback speed, or a change in voice associated with the face. Computer readable storage medium according to Claim 16 wherein the one or more processors are further configured to perform the operations including sending the video clip of the virtual avatar to another electronic device.

Images