CN115695684A - Multimedia data editing method and device, storage medium and electronic device - Google Patents

Abstract

The application discloses a multimedia data editing method, device, storage medium and electronic device, and relates to the field of smart home. The multimedia data editing method includes: extracting an initial image from the initial multimedia data collected from the scene object in the target scene Features and initial sound features; identify the target emotional intensity of the scene object according to the initial image features and initial sound features; when the target emotional intensity is greater than or equal to the target intensity threshold, intercept the target moment from the initial multimedia data. The target multimedia data, wherein the target moment is the moment corresponding to the identified multimedia data whose target emotional intensity is greater than or equal to the target intensity threshold. The problem of low efficiency in editing the multimedia data of the wonderful moment in the related technology is solved, and the technical effect of improving the efficiency of editing the multimedia data of the wonderful moment is realized.

Description

Multimedia data editing method, device, storage medium and electronic device

technical field

The present application relates to the field of smart home, in particular, to a multimedia data editing method, device, storage medium and electronic device.

Background technique

In the existing technology, when recording the wonderful moments in daily life, it is often necessary for the user to take the initiative to shoot and record through mobile phones, cameras, etc., such a recording method, on the one hand, may cause the user to miss the record because the user will not have time to shoot in the future Multimedia data of exciting moments; on the other hand, this method is to shoot under the subjective consciousness of the person being photographed, and the shooting effect is "deliberately" and may not be able to capture the excitement of the user in the most natural situation moment.

Aiming at the problems in related technologies, such as low efficiency of editing the multimedia data of wonderful moments, no effective solution has been proposed yet.

Contents of the invention

Embodiments of the present application provide a multimedia data editing method, device, storage medium, and electronic device, so as to at least solve the problems in related technologies such as low efficiency of editing multimedia data of wonderful moments.

According to one embodiment of the present application, a method for editing multimedia data is provided, including: extracting initial image features and initial sound features from initial multimedia data collected from scene objects in a target scene, wherein the initial image features It is used to represent the emotion of the scene object through the movement of the target object part on the scene object, and the initial sound feature is used to represent the emotion of the scene object through the sound emitted by the scene object;

Identifying the target emotional intensity of the scene object according to the initial image feature and the initial sound feature, wherein the target emotional intensity is used to indicate the intensity of the emotion of the scene object;

In the case where the target emotional strength is greater than or equal to the target strength threshold, intercept the target multimedia data within the target time period where the target moment is located from the initial multimedia data, wherein the target moment is when the target emotional strength is identified The moment corresponding to the multimedia data greater than or equal to the target intensity threshold.

Optionally, the identifying the target emotional intensity of the scene object according to the initial image features and the initial sound features includes:

performing feature fusion on the initial expression features and initial action features included in the initial image features, and the initial sound features to obtain initial fusion features, wherein the initial expression features are used to represent the facial activity of the scene object The emotion possessed by the scene object, the initial action feature is used to characterize the emotion possessed by the scene object through the physical activity of the scene object, and the initial fusion feature includes the emotion possessed by the scene object The features whose correlation degree is greater than the target threshold;

Determining the target emotional intensity of the scene objects matching the initial fusion features.

Optionally, the determining the target emotional intensity of the scene object matching the initial fusion feature includes:

determining the target performance severity of the scene object matched with the initial fusion feature, wherein the target performance severity is used to indicate the severity of the external performance of the scene object;

The target expressive intensity is converted to the target emotional intensity.

Optionally, the determining the target emotional intensity of the scene object matching the initial fusion feature includes:

The initial fusion feature is input into the target emotion recognition model, wherein the target emotion recognition model is obtained by using the object fusion feature marked with the emotional strength label to train the initial emotion recognition model;

Acquiring the target emotion intensity output by the target emotion recognition model.

Optionally, the intercepting the target multimedia data within the target time period at the target moment from the initial multimedia data includes one of the following:

Intercepting from the first moment to the target moment in the initial multimedia data to obtain the target multimedia data, wherein the first moment is the target time period before the target moment from the target moment time;

Intercepting from the target time to a second time in the initial multimedia data to obtain the target multimedia data, wherein the second time is the distance from the target time after the target time to the target time period time;

The target multimedia data is obtained by intercepting from the third moment to the fourth moment in the initial multimedia data, wherein the distance between the third moment and the fourth moment is the target time period, and the target moment is within the between the third moment and the fourth moment.

Optionally, after the intercepting the target multimedia data within the target time period at the target moment from the initial multimedia data, the method further includes:

Determining the target account corresponding to the scene object and the target emotion type corresponding to the target multimedia data, and obtaining the target data frame with the highest intensity of the target emotion from the target multimedia data;

storing the target multimedia data with the target data frame as the display cover in the data set corresponding to the target emotion type under the target account.

Optionally, after the intercepting the target multimedia data within the target time period at the target moment from the initial multimedia data, the method further includes:

Receiving a data call request, wherein the data call request is used to request to call multimedia data showing a target scene object conforming to the target keyword;

In response to the data call request, the target keywords are divided into scene attributes and emotional attributes;

Screening candidate multimedia data conforming to the scene attribute from the multimedia data set stored for the target scene object;

Screening multimedia data matching the emotional attribute from the candidate multimedia data as the multimedia data to be invoked;

Display the multimedia data to be called.

According to another embodiment of the present application, a device for editing multimedia data is also provided, including:

An extraction module, configured to extract initial image features and initial sound features from the initial multimedia data collected from the scene object in the target scene, wherein the initial image features are used to represent the motion of the target object part on the scene object. The emotion that the scene object has, the initial sound feature is used to represent the emotion that the scene object has through the sound sent by the scene object;

An identification module, configured to identify the target emotional intensity of the scene object according to the initial image feature and the initial sound feature, wherein the target emotional intensity is used to indicate the intensity of the emotion of the scene object;

An intercepting module, configured to intercept the target multimedia data within the target time period where the target moment is located from the initial multimedia data when the target emotional intensity is greater than or equal to the target intensity threshold, wherein the target moment is identified The moment corresponding to the multimedia data whose target emotional intensity is greater than or equal to the target intensity threshold.

According to yet another aspect of the embodiments of the present application, a computer-readable storage medium is also provided, and a computer program is stored in the computer-readable storage medium, wherein the computer program is set to execute edit method.

According to yet another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein the above-mentioned processor executes the above-mentioned A method for editing multimedia data.

In the embodiment of the present application, the initial image features and initial sound features are extracted from the initial multimedia data collected from the scene objects in the target scene, wherein the initial image features are used to represent the scene objects through the motion of the target object parts on the scene objects. emotion, the initial sound feature is used to represent the emotion of the scene object through the sound emitted by the scene object; the target emotional intensity of the scene object is identified according to the initial image feature and the initial sound feature, wherein the target emotional intensity is used to indicate the scene object The intensity of the emotion possessed; when the target emotional intensity is greater than or equal to the target intensity threshold, the target multimedia data within the target time period where the target moment is located is intercepted from the initial multimedia data, wherein the target moment is the identification of the target emotional intensity At the moment corresponding to the multimedia data greater than or equal to the target intensity threshold, the target emotional intensity of the scene object can be identified according to the initial image features and initial sound features extracted from the initial multimedia data. When the target emotional intensity is greater than or equal to the target intensity In the case of the threshold value, it can be indicated that there is a wonderful moment of the scene object at the target moment. In this case, the target multimedia data in the target time period of the target moment can be intercepted from the initial multimedia data, and the image combined with the scene object can be realized. Features and sound features, multi-dimensional recognition of the emotional intensity of the scene object, and can automatically record the multimedia data of the wonderful moment when the wonderful moment of the scene object is recognized. By adopting the above technical solution, the problem of low efficiency in editing the multimedia data of the wonderful moment in the related art is solved, and the technical effect of improving the efficiency of editing the multimedia data of the wonderful moment is realized.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description serve to explain the principles of the application.

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, In other words, other drawings can also be obtained from these drawings on the premise of not paying creative work.

Fig. 1 is a schematic diagram of a hardware environment according to a method for editing multimedia data according to an embodiment of the present application;

Fig. 2 is a flowchart of a method for editing multimedia data according to an embodiment of the present application;

FIG. 3 is a schematic diagram of storing target multimedia data according to an embodiment of the present application;

Fig. 4 is a schematic diagram of a registered face of a scene object according to an embodiment of the present application;

FIG. 5 is a schematic diagram of calling multimedia data according to an embodiment of the present application;

FIG. 6 is a schematic diagram of a method for editing multimedia data according to an embodiment of the present application;

Fig. 7 is a structural block diagram of an apparatus for editing multimedia data according to an embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiment of the application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the application. Obviously, the described embodiment is only It is an embodiment of a part of the application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

It should be noted that the terms "first" and "second" in the description and claims of the present application and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

According to one aspect of the embodiments of the present application, a method for editing multimedia data is provided. The multimedia data editing method is widely used in whole-house intelligent digital control application scenarios such as smart home, smart home, smart household equipment ecology, and intelligence house ecology. Optionally, in this embodiment, the above multimedia data editing method may be applied to a hardware environment composed of a terminal device 102 and a server 104 as shown in FIG. 1 . As shown in Figure 1, the server 104 is connected to the terminal device 102 through the network, and can be used to provide services (such as application services, etc.) for the terminal or the client installed on the terminal. To provide data storage services for the server 104, cloud computing and/or edge computing services may be configured on the server or independently of the server to provide data computing services for the server 104.

The foregoing network may include but not limited to at least one of the following: a wired network and a wireless network. The above-mentioned wired network may include but not limited to at least one of the following: wide area network, metropolitan area network, and local area network, and the above-mentioned wireless network may include but not limited to at least one of the following: WIFI (Wireless Fidelity, Wireless Fidelity), Bluetooth. The terminal device 102 is not limited to PC, mobile phone, tablet computer, smart air conditioner, smart hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing device, smart dishwasher, smart projection device , smart TV, smart drying rack, smart curtain, smart video, smart socket, smart audio, smart speaker, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart sweeping robot, smart window cleaning robot, smart mopping robot, Smart air purification equipment, smart steamer, smart microwave oven, smart kitchen treasure, smart purifier, smart water dispenser, smart door lock, etc.

In this embodiment, a method for editing multimedia data is provided, which is applied to the above-mentioned terminal device. FIG. 2 is a flow chart of a method for editing multimedia data according to an embodiment of the present application. As shown in FIG. 2 , the process includes Follow the steps below:

Step S202, extracting initial image features and initial sound features from the initial multimedia data collected from the scene objects in the target scene, wherein the initial image features are used to characterize the scene through the motion of the target object parts on the scene objects The emotion that the object has, and the initial sound feature is used to represent the emotion that the scene object has through the sound emitted by the scene object;

Step S204, identifying the target emotional intensity of the scene object according to the initial image feature and the initial sound feature, wherein the target emotional intensity is used to indicate the intensity of the emotion of the scene object;

Step S206, when the target emotional intensity is greater than or equal to the target intensity threshold, intercept the target multimedia data within the target time period where the target moment is located from the initial multimedia data, wherein the target moment is when the target moment is identified The moment corresponding to the multimedia data whose target emotional intensity is greater than or equal to the target intensity threshold.

Through the above steps, the target emotional intensity of the scene object can be identified according to the initial image features and initial sound features extracted from the initial multimedia data. When the target emotional intensity is greater than or equal to the target intensity threshold, it can be indicated that the target emotional intensity appears at the target moment. In this case, the target multimedia data within the target time period at the target moment can be intercepted from the initial multimedia data, realizing the multi-dimensional recognition of the scene object by combining the image features and sound features of the scene object emotional intensity, and can automatically record the multimedia data of the wonderful moment in the case of recognizing the wonderful moment of the scene object. By adopting the above technical solution, the problem of low efficiency in editing the multimedia data of the wonderful moment in the related art is solved, and the technical effect of improving the efficiency of editing the multimedia data of the wonderful moment is realized.

In the technical solution provided in the above step S202, it is possible but not limited to extract the initial image features representing the movement of the facial parts and body parts of the scene object, and the initial sound features representing the sound emitted by the scene object from the initial multimedia data, to achieve The influence of factors such as facial movement, body movement, and the sound emitted by the scene object on the emotion of the scene object is considered in multiple dimensions.

Optionally, in this embodiment, the target scene may include, but is not limited to, one or more scene objects, and the target scene may include, but is not limited to, family scenes, office scenes, or sports scenes, etc., and the scene objects may, but not limited to One or more family members in a family scene, or one or more office workers in an office scene, or one or more sports personnel included in a sports scene, and so on.

Optionally, in this embodiment, the initial multimedia data may include, but is not limited to, data such as video data, image data, and audio data recorded for the target scene, and may, but is not limited to, be transmitted through smart speakers, smart TVs, smart sweeping robots, and Smart audio and other smart devices with camera function and voice function can detect scene objects in the target scene in real time. Extracting initial image features and initial sound features from initial multimedia data efficiently improves the efficiency of extracting initial image features and initial sound features.

Optionally, in this embodiment, the initial image features may include, but are not limited to, the features of the facial movement of the scene object (may include, but are not limited to, the movement distance of the eyebrows, the upward distance of the corners of the mouth, the downward distance of the corners of the mouth, etc. ) and the features of the scene object’s body movement (which may include, but is not limited to, the lifting range of the leg, the swinging range of the arm, and the bending angle of the body, etc.), and the initial sound feature may include, but is not limited to, the sound emitted by the scene object It has the characteristics of amplitude, frequency and so on.

In the technical solution provided in the above step S204, in the case of extracting the initial image features and initial sound features from the initial multimedia data, it is possible, but not limited to, to comprehensively consider factors such as the initial image features and initial sound features on the scene object. The emotional intensity of the scene object and the intensity of the emotion of the scene object have realized the multi-dimensional recognition of the intensity of the emotion of the scene object, and the accuracy of the target emotional intensity has been improved.

Optionally, in this embodiment, the emotion of the scene object may include but not limited to hi (can include but not limited to liking, joy, liking and liking, etc.); Anger, resentment and resentment, etc.); mourning (can be but not limited to include sadness, sorrow, sorrow, pity, pity, mourning, pity, mourning, mourning, etc.); joy (can be but not limited to include joy, joy, happy and joyful, etc.); frightened (can be but not limited to include shock, astonishment, panic, palpitations, surprise, surprise, surprise and surprise, etc.); , fear, etc.); think (can but not be limited to include miss, miss and longing, etc.) and so on.

Optionally, in this embodiment, the scene objects may have the same or different emotional intensities, but are not limited to having the same or different emotional intensities. In the case that emotions with different degrees of severity have the same emotional strength, although the emotions of the scene objects have different degrees of severity, for the scene objects, some emotions may need to be severe to a certain extent. Has the value of recording (for example: sad emotions need to be recorded until they cry, and happy emotions will be recorded as soon as they appear, etc.), in this case, by assigning different degrees of intensity to the same emotion Intensity, which can meet the needs of users to filter the emotions that need to be recorded.

In an exemplary embodiment, the target emotional intensity of the scene object can be characterized but not limited to in the following way: performing feature fusion on the initial expression features and initial action features included in the initial image features, as well as the initial sound features , to obtain the initial fusion feature, wherein the initial expression feature is used to represent the emotion of the scene object through the facial activity of the scene object, and the initial action feature is used to represent the emotion of the scene object through the body activity of the scene object The emotions of the scene objects, the initial fusion features include features whose correlation with the emotions of the scene objects is greater than the target threshold; determine the features of the scene objects that match the initial fusion features Target emotional intensity.

Optionally, in this embodiment, in the case of performing feature fusion on the initial expression features, initial action features, and initial sound features, it is possible, but not limited, to start with the initial expression features, initial action features, and initial sound features. Filter out effective features (such as: the moving distance of the upper eyebrows on the face of the scene object in the initial expression features, the upward distance of the corners of the mouth and the downward distance of the corners of the mouth, etc. are effective features, and the color of the eyebrows on the face of the scene object, lips The lip color and so on are invalid features; in the initial action features, the range of motion of the legs, the swing range of the arms, and the bending angle of the body are valid features, while the skin color of the limbs, the fatness of the body, etc. are invalid features. ; In the initial sound feature, the amplitude, frequency and other features of the sound emitted by the scene object are effective features, while the accent of the scene object, etc. are invalid features). Then, the selected effective expression features, action features, and voice features are fused, so that the initial fusion features can accurately represent the intensity of the emotion of the scene object, and the accuracy of the initial fusion features is improved.

Optionally, in this embodiment, the initial expression features can be used, but not limited to, to represent the emotions of the scene object through the movements of the eyebrows, eyes, mouth, nose, and muscles on the face of the scene object, and the initial action features It can be but not limited to the limbs of the scene object (it can be but not limited to include human upper limbs and double lower limbs, the upper limbs can be but not limited to include shoulders, arms, elbows, forearms, wrists and hands, etc., the lower limbs can be but not limited to include Buttocks, thighs, knees, shins, and feet, etc.) represent the emotions that scene objects generally have. By fusing effective features from initial expression features, initial action features, and initial voice features, the accuracy of target emotional intensity is improved.

In an exemplary embodiment, the target emotion intensity may be determined in the following manner but not limited to: determining the target performance intensity of the scene object matching the initial fusion feature, wherein the target expression intensity is used to indicate The intensity of the external performance of the scene object; converting the intensity of the target performance into the intensity of the target emotion.

Optionally, in this embodiment, it is possible, but not limited to, to determine the severity of the external performance of the scene object according to the initial fusion features, and then convert the target performance severity into the emotional severity of the scene object's inner emotions to achieve In order to determine the intensity of the emotion of the scene object according to the actual external performance of the scene object, the accuracy of the target emotion intensity is improved.

Optionally, in this embodiment, the intensity of facial activities of the scene object can be, but not limited to, based on the range of motion, distance, direction, etc. Wait for confirmation. The intensity of the limb activity of the scene object may be determined according to, but not limited to, the range of motion, the distance of motion, the direction of motion, and the like of the limbs of the scene object. The severity of the sound emitted by the scene object may be determined according to, but not limited to, the loudness, amplitude, pitch, etc. of the sound emitted by the scene object. The intensity of the target performance can be determined based on, but not limited to, the intensity of the facial and body movements of the scene object and the sound of the scene object, which improves the accuracy of the intensity of the target performance.

In an exemplary embodiment, the target emotion intensity can be determined in the following manner, but not limited to: input the initial fusion features into the target emotion recognition model, wherein the target emotion recognition model uses object fusion marked with emotion intensity labels Features obtained by training an initial emotion recognition model; obtaining the target emotion intensity output by the target emotion recognition model.

Optionally, in this embodiment, the emotional intensity tag can include, but is not limited to, the degree of happiness (such as: super happy, very happy, generally happy, etc.), the degree of sadness (such as: very sad, a little sad, not sad etc.), and the degree of surprise (such as very surprised, a little surprised, not surprised at all, etc.) and so on.

Optionally, in this embodiment, but not limited to, the initial fusion feature may be input into the target emotion recognition model, and the result output by the target emotion recognition model may be obtained as the target emotion intensity. It realizes the rapid acquisition of the target emotional intensity corresponding to the initial fusion feature, and improves the efficiency of acquiring the target emotional intensity.

In the technical solution provided by the above step S206, when the target emotional intensity is greater than or equal to the target intensity threshold, it can be indicated that the scene object’s emotion is strong enough, and in this case, the identification can be intercepted from the initial multimedia data The multimedia data within the target time period corresponding to the time when the target emotional intensity is greater than or equal to the target intensity threshold is taken as the target multimedia data, realizing the recognition of the scene object in the comprehensive scene object's facial activities, body activities and the sound emitted by the scene object When the emotions are strong enough, the target multimedia data of the wonderful moment of the scene object can be automatically intercepted in time, avoiding missing the wonderful moment caused by not shooting the scene object in time, and improving the efficiency of recording multimedia data.

Optionally, in this embodiment, when the target emotional intensity is less than the target intensity threshold, it may indicate that the emotion of the scene object is not strong enough. In this case, the initial multimedia data may not be edited , but continue to collect the initial multimedia data, and extract the initial image features and initial sound features in the initial multimedia data in real time, and identify the target emotional intensity of the scene object according to the initial image feature and initial sound feature, until the target emotional intensity of the scene object is greater than Or if it is equal to the target strength threshold, then intercept the target multimedia data from the initial multimedia data.

Optionally, in this embodiment, when the target emotional intensity is greater than or equal to the target intensity threshold, it can be indicated that the emotion of the scene object is strong enough (which may include, but is not limited to, that the scene object is screaming, beating, Laughing, crying, fighting, or covering the stomach, turning on one's back, dancing, jumping and other strenuous movements of the human body, or expressions of surprise, sadness, fear, anger, happiness, disgust and contempt, etc.), in such cases, you can In the most natural state of the scene object, the most realistic multimedia data of the scene object is automatically recorded. Avoiding the recording of all multimedia data greatly saves storage space, and saves the subsequent process of selecting wonderful multimedia data, greatly improving the efficiency of recording multimedia data.

In an exemplary embodiment, intercepting the target multimedia data within the target time period at the target moment from the initial multimedia data may include, but is not limited to, one of the following situations:

Situation 1, the target multimedia data is obtained by intercepting from the first moment in the initial multimedia data to the target moment, wherein the first moment is before the target moment and the distance from the target moment is the target moment moment of time.

Optionally, in this embodiment, multimedia data between the first moment before the target moment and the target moment may be intercepted from the initial multimedia data, but not limited to, as the target multimedia data. In this case, the scene object has The intensity of the emotion may be, but not limited to, reaching the target intensity threshold at the target moment. In this way, the multimedia data including the process in which the intensity of the emotion of the scene object reaches the target intensity threshold can be intercepted.

Scenario 2: The target multimedia data is obtained by intercepting from the target time to a second time in the initial multimedia data, wherein the second time is after the target time and the distance from the target time is the target time moment of time.

Optionally, in this embodiment, multimedia data between the target moment and the second moment after the target moment may be intercepted from the initial multimedia data, but not limited to, as the target multimedia data. In this case, the scene object has The intensity of the emotion can be but not limited to reaching the target intensity threshold at the target moment, and the intensity of the emotion of the scene object may continue to increase or decrease or remain unchanged after the target moment. This method can be intercepted The multimedia data of the process of changing the intensity of the emotion of the scene object after the intensity of the emotion of the scene object reaches the target intensity threshold.

Situation 3, the target multimedia data is obtained by intercepting from the third moment to the fourth moment in the initial multimedia data, wherein the distance between the third moment and the fourth moment is the target time period, and the target A time is between said third time and said fourth time.

Optionally, in this embodiment, when the intensity of the emotion of the scene object reaches the target intensity threshold at the target moment, the initial multimedia data may be intercepted from the third moment before the target moment to The multimedia data between the fourth moment after the target moment is used as the target multimedia data. This interception method can completely record the emotional intensity of the scene object before reaching the target intensity threshold and the emotional intensity of the scene object reaching the target. The process after the intensity threshold.

In an exemplary embodiment, the target multimedia data may be stored in the following manner but not limited to: determining the target account corresponding to the scene object and the target emotion type corresponding to the target multimedia data, and Obtain the target data frame with the highest intensity of the target emotion in the data; store the target multimedia data with the target data frame as the display cover in the data set corresponding to the target emotion type under the target account.

Optionally, in this embodiment, each scene object may have, but is not limited to, a corresponding account, and the account of each scene object may include, but is not limited to, data sets corresponding to various emotion types. In the case of determining the target account corresponding to the scene object and the target emotion type of the target multimedia data, if the data set corresponding to the target emotion type already exists, it is possible but not limited to directly store the target multimedia data in the target account under the target emotion type in the corresponding data set. If it is the first time to store the target multimedia data of the target emotion type, in this case, but not limited to, the data set corresponding to the target emotion type may be created first, and then the target multimedia data is stored in the data set corresponding to the target emotion type.

Fig. 3 is a schematic diagram of storing target multimedia data according to an embodiment of the present application. As shown in Fig. 3, the target account corresponding to the scene object may include, but is not limited to, emotion type 1, emotion type 2, ..., emotion Type m, ..., data set 1 of emotion type n, data set 2, ..., data set m, ..., data set n, the target emotion type corresponding to the target multimedia data can be but not limited to emotion type m, It is possible but not limited to store the target multimedia data in the data set m corresponding to the emotion type m. In addition, the multimedia data can also be encrypted and stored, and the multimedia data can be stored according to the storage method desired by the user (including but not limited to overwriting when the disk is full or stopping when the disk is full, etc.).

Optionally, in this embodiment, the target data frame with the highest target emotional intensity obtained from the target multimedia data may represent that the intensity of the scene object’s emotion has reached its peak. In this case, it may indicate that the target data frame It is the most exciting frame in the target multimedia data. By using the most exciting frame in the multimedia data of each scene object as the cover of the multimedia data, it is convenient for the user to directly view the most exciting moment in the multimedia data and improve the user's Check out the experience. In addition, the cover of each scene object can also be exported to create a photo wall of wonderful moments, emoticons, family image collections, etc., providing users with a smart and interesting experience.

In an exemplary embodiment, the multimedia data can be invoked in the following manner, but not limited to: receiving a data invocation request, wherein the data invocation request is used to request invocation of the multimedia data showing the target scene object that meets the target keyword; responding The data call request divides the target keywords into scene attributes and emotional attributes; screens candidate multimedia data that conforms to the scene attributes from the multimedia data collection stored for the target scene object; selects from the candidate multimedia data Screening the multimedia data matching the emotional attribute as the multimedia data to be called; displaying the multimedia data to be called.

Optionally, in this embodiment, before invoking the multimedia data, the scene object in the target scene may, but not limited to, perform face registration first. FIG. 4 is a face registration of a scene object according to an embodiment of the present application Schematic diagram, as shown in Figure 4, family members (ie, the above-mentioned scene objects) in the family scene (ie, the above-mentioned target scene) can, but not limited to, enter their own age, name, etc. in front of a smart device with a camera. Smart devices will collect images of scene objects, and then enter the face information of family members to build a family face information database.

Optionally, in this embodiment, in the case where multimedia data needs to be edited, face recognition may be performed first, but not limited to, if the face is identified as a face recorded in the family face information database, Only then can the stored multimedia data be viewed, selected, exported, etc., and FIG. 5 is a schematic diagram of calling multimedia data according to an embodiment of the present application. As shown in FIG. That is, the smart device with camera function and voice function in the above-mentioned target scene) performs image collection and sound collection (that is, the above-mentioned collection of initial multimedia data), which can be, but not limited to, combined with registered family members recorded in the family face information database The face information of the family member of entry, carry out face recognition; And carry out motion recognition and voice recognition; In the case of normal viewing, registration, or deletion of image data, user authentication and login are required first, and the account that has passed user authentication can perform normal viewing, registration, or deletion of stored image data, etc. Editing operations. In this way, the security of the multimedia data can be ensured, and the leakage of the user's multimedia data can be prevented from being viewed by anyone. And when the smart device storing the multimedia data collection fails or needs to be sold or discarded, the binding between the smart device and the user's account can be cancelled, or all image data can be cleared to ensure family privacy Safety.

Optionally, in this embodiment, the user can, but is not limited to, edit the target keyword of the multimedia data that he wants to view on the visual interface, and can, but is not limited to, divide the target keyword into the time and place of intercepting the multimedia data (i.e. The above-mentioned target scene), members (that is, the above-mentioned scene object), and the number of people (that is, the number of the above-mentioned scene object) and other scene attributes, and emotional attributes such as expressions, actions, and voices.

Optionally, in this embodiment, it is possible, but not limited to, first to filter candidate multimedia data conforming to scene attributes from the multimedia data set stored for the target scene object; and then to filter multimedia data matching emotional attributes from candidate multimedia data as Multimedia data to be called. Alternatively, firstly select the multimedia data matching the emotional attribute from the multimedia data set stored for the target scene object; and then select the multimedia data matching the scene attribute from the multimedia data matching the emotional attribute as the multimedia data to be called.

In order to better understand the editing process of the above multimedia data, the editing flow of the above multimedia data will be described below in conjunction with an optional embodiment, but it is not used to limit the technical solution of the embodiment of the present application.

In this embodiment, a method for editing multimedia data is provided. FIG. 6 is a schematic diagram of a method for editing multimedia data according to an embodiment of the present application. As shown in FIG. 6 , it can be deployed in a home scene ( That is, smart devices such as smart air conditioners, smart TVs, and smart speakers in the above-mentioned target scene) collect initial multimedia data of scene objects in real time, and input the collected initial multimedia data into the face feature recognition model, action feature recognition model and voice The feature recognition model identifies the initial expression features, initial action features, and initial voice features in the initial multimedia data; it can, but is not limited to, input the initial expression features, initial activity features, and initial voice features into the feature fusion model for feature fusion to obtain the initial fusion Features; can be but not limited to input the initial fusion feature into the emotion recognition model, identify the emotion corresponding to the initial fusion feature, and obtain the emotion of the scene object; can but not be limited to determine the target emotion intensity of the emotion of the scene object; in the target When the emotional intensity is greater than or equal to the target intensity threshold, intercept and identify the target multimedia data at the moment corresponding to the multimedia data whose target emotional intensity is greater than or equal to the target intensity threshold from the initial multimedia data; when the target emotional intensity is less than the target intensity threshold In this case, the initial multimedia data is continuously collected until it is determined that the target emotional intensity is greater than or equal to the target intensity threshold, and the time corresponding to the multimedia data identifying that the target emotional intensity is greater than or equal to the target intensity threshold is intercepted from the initial multimedia data. Target multimedia data.

It should be noted that in Figure 6, the initial multimedia data collection is only performed through smart air conditioners, smart TVs, and smart speakers. Speakers, smart TVs, smart speakers, smart air conditioners and other smart devices with camera and voice functions are used to collect initial multimedia data. This application is not limited to this.

Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product in essence or the part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to make a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) execute the method of each embodiment of the present application.

Fig. 7 is a structural block diagram of an editing device for multimedia data according to an embodiment of the present application, as shown in Fig. 7 , including:

The extraction module 72 is used to extract initial image features and initial sound features from the initial multimedia data collected from the scene objects in the target scene, wherein the initial image features are used to represent the movement of the target object parts on the scene objects The emotion that the scene object has, and the initial sound feature is used to represent the emotion that the scene object has through the sound emitted by the scene object;

An identification module 74, configured to identify the target emotional intensity of the scene object according to the initial image feature and the initial sound feature, wherein the target emotional intensity is used to indicate the intensity of the emotion of the scene object;

The intercepting module 76 is used to intercept the target multimedia data within the target time period where the target moment is located from the initial multimedia data when the target emotional intensity is greater than or equal to the target intensity threshold, wherein the target moment is the identification The moment corresponding to the multimedia data whose target emotional intensity is greater than or equal to the target intensity threshold is displayed.

Through the above-mentioned embodiments, the target emotional intensity of the scene object can be identified according to the initial image features and initial sound features extracted from the initial multimedia data, and when the target emotional intensity is greater than or equal to the target intensity threshold, it can be indicated that at the target moment There is a wonderful moment of the scene object. In this case, the target multimedia data within the target time period at the target time can be intercepted from the initial multimedia data, and the multi-dimensional recognition of the scene object can be achieved by combining the image features and sound features of the scene object. emotional intensity, and can automatically record the multimedia data of the wonderful moment in the case of recognizing the wonderful moment of the scene object. By adopting the above technical solution, the problem of low efficiency in editing the multimedia data of the wonderful moment in the related art is solved, and the technical effect of improving the efficiency of editing the multimedia data of the wonderful moment is realized.

Optionally, the identification module includes:

A fusion unit, configured to perform feature fusion on the initial expression features and initial action features included in the initial image features, as well as the initial sound features, to obtain initial fusion features, wherein the initial expression features are used to pass through the scene The facial activity of the object represents the emotion of the scene object, and the initial action feature is used to represent the emotion of the scene object through the body activity of the scene object, and the initial fusion feature includes the The feature that the correlation degree between the emotions of the object is greater than the target threshold;

A determining unit, configured to determine the target emotional intensity of the scene object that matches the initial fusion feature.

Optionally, the determining unit is used for:

determining the target performance severity of the scene object matched with the initial fusion feature, wherein the target performance severity is used to indicate the severity of the external performance of the scene object;

The target expressive intensity is converted to the target emotional intensity.

Optionally, the determining unit is used for:

The initial fusion feature is input into the target emotion recognition model, wherein the target emotion recognition model is obtained by using the object fusion feature marked with the emotional strength label to train the initial emotion recognition model;

Acquiring the target emotion intensity output by the target emotion recognition model.

Optionally, the interception module includes one of the following:

A first intercepting unit, configured to intercept from the first moment to the target moment in the initial multimedia data to obtain the target multimedia data, wherein the first moment is a distance from the target moment before the target moment is the moment of the target time period;

The second intercepting unit is configured to intercept from the target time to a second time in the initial multimedia data to obtain the target multimedia data, wherein the second time is a distance from the target time after the target time is the moment of the target time period;

The third intercepting unit is configured to intercept from the third moment to the fourth moment in the initial multimedia data to obtain the target multimedia data, wherein the distance between the third moment and the fourth moment is the target time period , the target moment is between the third moment and the fourth moment.

Optionally, the device also includes:

A determination module, configured to determine the target account corresponding to the scene object and the target account corresponding to the target multimedia data after the target multimedia data within the target time period at the target time is intercepted from the initial multimedia data. emotion type, and obtain the target data frame with the highest intensity of the target emotion from the target multimedia data;

A storage module, configured to store the target multimedia data with the target data frame as the display cover in the data set corresponding to the target emotion type under the target account.

Optionally, the device also includes:

The receiving module is configured to receive a data invocation request after intercepting the target multimedia data in the target time period at the target moment from the initial multimedia data, wherein the data invocation request is used to request invoking to show that the key to meet the target The multimedia data of the target scene object of the word;

A division module, configured to divide the target keywords into scene attributes and emotional attributes in response to the data call request;

A first screening module, configured to screen candidate multimedia data conforming to the scene attribute from the multimedia data set stored for the target scene object;

The second screening module is used to screen the multimedia data matching the emotional attribute from the candidate multimedia data as the multimedia data to be invoked;

A display module, configured to display the multimedia data to be called.

An embodiment of the present application further provides a storage medium, the storage medium includes a stored program, wherein the above-mentioned program executes any one of the above-mentioned methods when running.

Optionally, in this embodiment, the above-mentioned storage medium may be configured to store program codes for performing the following steps:

S1. Extracting initial image features and initial sound features from the initial multimedia data collected on the scene objects in the target scene, wherein the initial image features are used to characterize the scene objects through the motion of the target object parts on the scene objects The emotion possessed, the initial sound feature is used to represent the emotion possessed by the scene object through the sound emitted by the scene object;

S2. Identify the target emotional intensity of the scene object according to the initial image feature and the initial sound feature, wherein the target emotional intensity is used to indicate the intensity of the emotion of the scene object;

S3. In the case that the target emotional intensity is greater than or equal to the target intensity threshold, intercept the target multimedia data within the target time period where the target moment is located from the initial multimedia data, wherein the target moment is when the target is identified The moment corresponding to the multimedia data whose emotional intensity is greater than or equal to the target intensity threshold.

An embodiment of the present application also provides an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments.

Optionally, the above-mentioned electronic device may further include a transmission device and an input-output device, wherein the transmission device is connected to the above-mentioned processor, and the input-output device is connected to the above-mentioned processor.

Optionally, in this embodiment, the above-mentioned processor may be configured to execute the following steps through a computer program:

S1. Extracting initial image features and initial sound features from the initial multimedia data collected on the scene objects in the target scene, wherein the initial image features are used to characterize the scene objects through the motion of the target object parts on the scene objects The emotion possessed, the initial sound feature is used to represent the emotion possessed by the scene object through the sound emitted by the scene object;

S2. Identify the target emotional intensity of the scene object according to the initial image feature and the initial sound feature, wherein the target emotional intensity is used to indicate the intensity of the emotion of the scene object;

S3. In the case that the target emotional intensity is greater than or equal to the target intensity threshold, intercept the target multimedia data within the target time period where the target moment is located from the initial multimedia data, wherein the target moment is when the target is identified The moment corresponding to the multimedia data whose emotional intensity is greater than or equal to the target intensity threshold.

Optionally, in this embodiment, the above-mentioned storage medium may include but not limited to: U disk, read-only memory (Read-Only Memory, ROM for short), random access memory (Random Access Memory, RAM for short), Various media that can store program codes such as removable hard disks, magnetic disks, or optical disks.

Optionally, for specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiments and optional implementation manners, and details are not repeated in this embodiment.

Obviously, those skilled in the art should understand that each module or each step of the above-mentioned application can be realized by a general-purpose computing device, and they can be concentrated on a single computing device, or distributed in a network composed of multiple computing devices Alternatively, they may be implemented in program code executable by a computing device so that they may be stored in a storage device to be executed by a computing device, and in some cases in an order different from that shown here The steps shown or described are carried out, or they are separately fabricated into individual integrated circuit modules, or multiple modules or steps among them are fabricated into a single integrated circuit module for implementation. As such, the present application is not limited to any specific combination of hardware and software.

The above description is only the preferred embodiment of the present application. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present application, some improvements and modifications can also be made. These improvements and modifications are also It should be regarded as the protection scope of this application.

Claims (10)

1. A method for editing multimedia data, characterized in that, comprising: Extracting initial image features and initial sound features from the initial multimedia data collected on the scene object in the target scene, wherein the initial image features are used to characterize the scene object through the motion of the target object part on the scene object. emotion, the initial sound feature is used to represent the emotion of the scene object through the sound emitted by the scene object; Identifying the target emotional intensity of the scene object according to the initial image feature and the initial sound feature, wherein the target emotional intensity is used to indicate the intensity of the emotion of the scene object; In the case where the target emotional strength is greater than or equal to the target strength threshold, intercept the target multimedia data within the target time period where the target moment is located from the initial multimedia data, wherein the target moment is when the target emotional strength is identified The moment corresponding to the multimedia data greater than or equal to the target intensity threshold. 2. The method according to claim 1, wherein the identifying the target emotional intensity of the scene object according to the initial image feature and the initial sound feature comprises: performing feature fusion on the initial expression features and initial action features included in the initial image features, and the initial sound features to obtain initial fusion features, wherein the initial expression features are used to represent the facial activity of the scene object The emotion possessed by the scene object, the initial action feature is used to characterize the emotion possessed by the scene object through the physical activity of the scene object, and the initial fusion feature includes the emotion possessed by the scene object The features whose correlation degree is greater than the target threshold; Determining the target emotional intensity of the scene objects matching the initial fusion features. 3. The method according to claim 2, wherein the determination of the target emotional intensity of the scene object matched with the initial fusion feature comprises: determining the target performance severity of the scene object matched with the initial fusion feature, wherein the target performance severity is used to indicate the severity of the external performance of the scene object; The target expressive intensity is converted to the target emotional intensity. 4. The method according to claim 2, wherein the determination of the target emotional intensity of the scene object matched with the initial fusion feature comprises: The initial fusion feature is input into the target emotion recognition model, wherein the target emotion recognition model is obtained by using the object fusion feature marked with the emotional strength label to train the initial emotion recognition model; Acquiring the target emotion intensity output by the target emotion recognition model. 5. The method according to claim 1, wherein the intercepting the target multimedia data in the target time period at the target moment from the initial multimedia data includes one of the following: Intercepting from the first moment to the target moment in the initial multimedia data to obtain the target multimedia data, wherein the first moment is the target time period before the target moment from the target moment time; Intercepting from the target time to a second time in the initial multimedia data to obtain the target multimedia data, wherein the second time is the distance from the target time after the target time to the target time period time; The target multimedia data is obtained by intercepting from the third moment to the fourth moment in the initial multimedia data, wherein the distance between the third moment and the fourth moment is the target time period, and the target moment is within the between the third moment and the fourth moment. 6. The method according to claim 1, wherein, after said intercepting the target multimedia data in the target time period where the target moment is located from the initial multimedia data, the method further comprises: Determining the target account corresponding to the scene object and the target emotion type corresponding to the target multimedia data, and obtaining the target data frame with the highest intensity of the target emotion from the target multimedia data; storing the target multimedia data with the target data frame as the display cover in the data set corresponding to the target emotion type under the target account. 7. The method according to claim 1, wherein, after said intercepting the target multimedia data in the target time period where the target moment is located from the initial multimedia data, the method further comprises: Receiving a data call request, wherein the data call request is used to request to call multimedia data showing a target scene object conforming to the target keyword; In response to the data call request, the target keywords are divided into scene attributes and emotional attributes; Screening candidate multimedia data conforming to the scene attribute from the multimedia data set stored for the target scene object; Screening multimedia data matching the emotional attribute from the candidate multimedia data as the multimedia data to be invoked; Display the multimedia data to be called. 8. An editing device for multimedia data, characterized in that, comprising: An extraction module, configured to extract initial image features and initial sound features from the initial multimedia data collected from the scene object in the target scene, wherein the initial image features are used to represent the motion of the target object part on the scene object. The emotion that the scene object has, the initial sound feature is used to represent the emotion that the scene object has through the sound sent by the scene object; An identification module, configured to identify the target emotional intensity of the scene object according to the initial image feature and the initial sound feature, wherein the target emotional intensity is used to indicate the intensity of the emotion of the scene object; An intercepting module, configured to intercept the target multimedia data within the target time period where the target moment is located from the initial multimedia data when the target emotional intensity is greater than or equal to the target intensity threshold, wherein the target moment is identified The moment corresponding to the multimedia data whose target emotional intensity is greater than or equal to the target intensity threshold. 9. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a stored program, wherein the method according to any one of claims 1 to 7 is executed when the program runs. 10. An electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute any one of claims 1 to 7 through the computer program. Methods.

Images