CN114005079B - Multimedia stream processing method and device - Google Patents

Abstract

The application provides a multimedia stream processing method and device. Wherein the method comprises the following steps: acquiring a multimedia stream fragment; decoding to obtain a video stream sub-segment and an audio stream sub-segment; analyzing the video stream sub-segment to generate scene information and first text information; analyzing the audio stream sub-segment to generate second text information; and processing the scene information, the first text information and the second text information to form an analysis abstract of the multimedia stream. Through disassembling the multimedia stream file, the content of the multimedia file can be identified in a complex scene by effectively combining various independent AI modules, and the identification efficiency of the existing independent AI technology in the complex scene is effectively improved.

Description

Multimedia stream processing method and device

technical field

The present application relates to the technical field of multimedia information identification, and in particular, to a method and device for processing a multimedia stream.

Background technique

With the continuous development and popularization of AI technology, many mature AI modules have appeared in the market, such as Alibaba Multimedia AI, which can be used to process information flow in media. For example, a video stream, an audio stream, or an information stream in which a video stream and an audio stream are combined in multimedia. In the process of processing these multimedia streams, the corresponding content in the acquired multimedia files can be identified through the corresponding AI module.

In the process of realizing the prior art, the inventors found that:

At present, the common AI module identification method is single. In the face of complex scenes to be recognized, it cannot be analyzed by a single AI module, thus reducing the recognition efficiency of multimedia files.

Therefore, it is necessary to provide a multimedia stream processing method and apparatus to solve the technical problem of low recognition efficiency of the existing independent AI technology in complex scenarios.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a multimedia stream processing method and apparatus, which are used to solve the technical problem of low recognition efficiency of the existing independent AI technology in complex scenarios.

Specifically, a multimedia stream processing method includes the following steps:

Get multimedia stream segments;

Decode to obtain video stream sub-segments and audio stream sub-segments;

Analyzing the sub-segments of the video stream to generate scene information and first text information;

analyzing the audio stream sub-segments to generate second textual information;

The scene information, the first text information and the second text information are processed to form an analysis summary of the multimedia stream.

Further, analyzing the sub-segments of the video stream to generate scene information, specifically including:

Analyze the sub-segments of the video stream to generate object-oriented identification information and description information of the object's action behavior.

Further, analyzing the sub-segments of the video stream to generate the first text information specifically includes:

Analyze the sub-segments of the video stream, and generate the first text information pointed to by the action behavior of the object.

Further, analyze the sub-segments of the video stream, and generate the first text information pointed to by the action behavior of the object, which specifically includes:

Analyze video stream sub-segments to obtain images that last for a preset duration;

The image is recognized using OCR to generate first text information.

Further, the first text information includes at least one of teaching link information and knowledge point information.

Further, the second text information specifically includes at least one of text error correction information, keyword information, question information, and emotion description information.

Further, processing the scene information, the first text information and the second text information to form an analysis summary of the multimedia stream, specifically including:

Cross-validation is performed on the scene information, the first text information and the second text information to form an analysis summary of the multimedia stream.

The embodiments of the present application also provide a multimedia stream processing apparatus.

Specifically, a multimedia stream processing device, comprising:

an acquisition module for acquiring multimedia stream segments;

The decoding module is used to decode and obtain the sub-segments of the video stream and the sub-segments of the audio stream;

a video analysis module, configured to analyze the sub-segments of the video stream to generate scene information and first text information;

an audio analysis module for analyzing the sub-segments of the audio stream to generate second text information;

An analysis summary generating module is configured to process the scene information, the first text information and the second text information to form an analysis summary of the multimedia stream.

Further, the video analysis module is used to analyze the sub-segments of the video stream to generate scene information, and is specifically used for:

Analyze the sub-segments of the video stream to generate object-oriented identification information and description information of the object's action behavior.

Further, the video analysis module is used to analyze the sub-segments of the video stream to generate the first text information, and is specifically used for:

Analyze the sub-segments of the video stream, and generate the first text information pointed to by the action behavior of the object.

The technical solutions provided by the application examples have at least the following beneficial effects:

By dismantling the multimedia stream files, it is possible to effectively combine various independent AI modules for multimedia file content recognition in complex scenarios, effectively improving the recognition efficiency of existing independent AI technologies in complex scenarios.

Description of drawings

The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the attached image:

FIG. 1 is a flowchart of a multimedia stream processing method provided by an embodiment of the present application.

FIG. 2 is a schematic structural diagram of a multimedia stream processing apparatus according to an embodiment of the present application.

100 Multimedia Streaming Device

11 Get modules

12 Decoding module

13 Video Analysis Module

14 Audio Analysis Module

15 Analysis summary generation module.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below with reference to the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

It can be understood that, the multimedia stream file records video stream information and audio stream information. Wherein, the video stream information mainly corresponds to several consecutive frames of images in the multimedia file; the audio stream information corresponds to a collection of voice information in the multimedia file. It can be seen that the video stream information correspondingly records scene information and text information related to the environment; the audio stream information records correspondingly the voice information corresponding to the relevant environment in the video stream. The scene information here can be understood as object information related to the presented object recorded in each frame of image; the text information here can be understood as the text-related symbol information recorded in each frame of image.

Through a single AI module, the scene information or text information in the video stream, or the audio stream information can be recognized in a single way, so as to identify the object behavior or existing text information in the video file, or the voice in the audio file. content. However, multimedia files in complex scenarios often contain both video information and audio information. If the identification continues through a single AI module, the video stream information and audio stream information recorded in the multimedia file cannot be fully identified. In this way, there is a certain error between the recognized target content and the real content recorded in the multimedia file. Although several different single AI modules can also be used to simultaneously identify the recorded content in complex scenarios, each single AI module has a large computational workload. In this way, the recognition speed of the multimedia file is reduced, and it is not conducive to the structural merging of the related recognition results.

An embodiment of the present application provides a multimedia stream processing method, which is mainly used for processing multimedia files in complex scenarios. In a specific implementation manner provided by this application, the multimedia stream processing method can be used to process a multimedia file that records a complex scene of a classroom teaching process. Specifically, please refer to FIG. 1, a multimedia stream processing method, comprising the following steps:

S100: Acquire a multimedia stream segment.

The multimedia stream segment here can be understood as a file that records media information such as text, graphics, video, animation, sound, and video of the corresponding scene. In a specific implementation manner provided by the present application, the acquired multimedia stream segment is a multimedia file with a certain duration and recorded with a classroom teaching scene. The multimedia stream segment can be obtained by shooting with a corresponding video shooting device. In this way, a real-time classroom scene can be photographed, thereby obtaining a multimedia file recording information such as voice, text, pictures, personnel objects, etc. in the course of classroom teaching.

S200: Decode to obtain video stream sub-segments and audio stream sub-segments.

The video stream sub-segment here can be understood as the image information in the multimedia segment. The audio stream sub-segment here can be understood as the sound information in the multimedia segment. Decode the acquired media stream segment, that is, extract the image information and sound information in the multimedia file with a certain duration, and convert it into several consecutive frames of images and continuous audio in the preset file format, so as to obtain the corresponding multimedia stream segment. Video stream subclip and audio stream subclip.

When the acquired multimedia stream segment is a multimedia file with a certain duration and recorded with classroom teaching scenes, after decoding, correspondingly obtained video stream sub-segments that record information such as words, pictures, personnel objects in the classroom teaching process, and record with Audio stream sub-segments of sound information during classroom teaching.

S310: Analyze the sub-segments of the video stream to generate scene information and first text information.

It can be understood that several consecutive frames of images in a multimedia stream segment can constitute a video stream sub-segment. Each frame of image is recorded with corresponding scene information. In a classroom teaching scenario, the video stream sub-segments are several consecutive frames of images that record information such as words, pictures, personnel objects, etc. in the course of classroom teaching. After the analysis of the AI module with corresponding functions, the specific scene information corresponding to the human object in the sub-segment of the current video stream, and the specific text information corresponding to the words and pictures in the sub-segment of the current video stream can be obtained.

Specifically, by identifying the human object in the sub-segment of the current video stream, the specific action category of the current human object can be determined, thereby facilitating the determination of the specific classroom teaching scene corresponding to the sub-segment of the current video stream. By identifying the specific text information corresponding to the text and picture in the sub-segment of the current video stream, the specific text type or description content corresponding to the text and picture information corresponding to the sub-segment of the current video stream can be determined, so that the corresponding sub-segment of the current video stream can be generated. the first text message. The first text information here can be understood as text information generated according to the sub-segments of the video stream.

S320: Analyze the audio stream sub-segments to generate second text information.

It can be understood that the audio stream sub-segment can be generated according to the voice information in the multimedia stream segment. In a classroom teaching scenario, the audio stream sub-segments are files that record sound information related to the classroom teaching process. The AI module with corresponding functions can determine the specific narration content of the audio stream sub-segment, and obtain second text information corresponding to the narration content of the audio stream sub-segment. The second text information here can be understood as text information generated according to the sub-segments of the audio stream.

S400: Process the scene information, the first text information and the second text information to form an analysis summary of the multimedia stream.

The analysis summary here can be understood as an overview of the specific real-time scene corresponding to the currently processed multimedia stream segment. The processing of the scene information, the first text information and the second text information is mainly to identify the key data with a high degree of correlation between the real-time scene corresponding to the multimedia stream segment. By integrating the identified target data, the specific teaching process corresponding to the currently processed multimedia stream segment can be obtained. Splitting multimedia stream segments into video stream sub-segments and audio stream sub-segments, and analyzing them through AI modules with corresponding functions, can effectively reduce the amount of data processed by an AI module with a single function, and can accurately select the ones with corresponding analysis functions. Analysis module, thereby improving the identification efficiency of multimedia stream segments.

Further, in a preferred embodiment provided by this application, analyzing the sub-segments of the video stream to generate scene information specifically includes: analyzing the sub-segments of the video stream, generating object-oriented identity feature identification information and descriptions of the object's action behavior information.

Here, the identity feature identification information of the object can be understood as the facial feature information of the object. It can be understood that the specific identity information of the person object can be determined by acquiring an image with the face of a certain person object and recognizing the facial features of the person object through a pre-trained recognition algorithm. For example, determine information such as the name and student ID of a certain student, or information such as the name and job ID of a certain teacher.

The description information of the action behavior of the object here can be understood as the specific action category of the person object in the sub-segment of the current video stream. It can be understood that, by acquiring an image with a body movement of a certain human object, and recognizing the image by a pre-trained recognition algorithm, the specific action category of the human object can be determined. For example, it is determined that the current action behavior of the person object is writing behavior, standing up behavior, or writing on the blackboard.

By identifying the specific identity information and behavior information of the objects involved in the sub-segments of the video stream, the behavior of teachers and students in the teaching scene corresponding to the sub-segments of the video stream can be determined, thereby improving the accuracy of the multimedia stream analysis summary.

Further, in a preferred embodiment provided by the present application, analyzing the sub-segments of the video stream to generate the first text information specifically includes: analyzing the sub-segments of the video stream to generate the first text information pointed to by the action behavior of the object.

The first text information pointed to by the action behavior of the object here can be understood as text information having a certain degree of correlation with the specific behavior of the person object. It can be understood that the first text information generated according to the sub-segments of the video stream includes specific text types or description contents corresponding to the text and picture information in the classroom teaching scenario. For example, the PPT display page in the classroom, the blackboard report information related to the classroom background. In the classroom teaching scenario, the text information in the PPT display page in the classroom is the text information related to the action behavior of the person and object. However, the information reported on the blackboard is background information in the sub-segment of the video stream, and has nothing to do with the actions and behaviors of people in the current scene, so it does not belong to the first text information.

By analyzing the first text information related to the actions of people and objects in the sub-segments of the video stream, the data processing amount of the corresponding functional modules can be reduced, and the recognition accuracy of the corresponding functional modules can be increased, thereby effectively improving the first text information. analytical efficiency.

Further, in a preferred embodiment provided by the present application, analyzing the sub-segments of the video stream to generate the first text information pointed to by the action behavior of the object specifically includes: analyzing the sub-segments of the video stream, and obtaining images that last for a preset duration; using The OCR identifies the image to generate first text information.

It can be understood that the first text information that the object action behavior points to is text information that has a certain degree of correlation with the object behavior in the scene corresponding to the current video stream sub-segment. In the classroom teaching scenario, the text information that has a certain degree of relevance with the behavior of the object can be understood as the text information related to the teaching content. For example, the blackboard written by the teacher, the text information in the PPT display page, etc. It should be pointed out that, in the actual teaching process, if the text information pointed to by the object's action behavior is more important, the human object will continue for a certain period of time when the relevant behavior of the corresponding text information is carried out. That is, images with important textual information stay longer. When the text information pointed to by the action behavior of the object is less important or worthless, it has a correspondingly shorter duration. That is, images with unimportant text information have shorter dwell time. Therefore, the importance of the text in the image can be judged according to the duration of the image. The image duration here can be preset according to actual conditions or empirical values. If the duration of a certain frame of image satisfies the preset condition, the subsequent text recognition process can be started. Correspondingly, if the duration of a certain frame of image does not meet the preset condition, it means that the corresponding text is of low importance, and corresponding identification does not need to be carried out. In this way, the recognition efficiency of the first text can be increased on the basis of ensuring the recognition accuracy of the first text.

Specifically, when recognizing the image memorization text information that satisfies the preset duration, the OCR recognition method can be used. That is, the optical character recognition technology is used to recognize the character information in the images satisfying the preset conditions in the sub-segments of the video stream. For example, the relevant content in the PPT page that satisfies the preset condition in the classroom display PPT is identified as the first text information by the OCR identification method. Alternatively, the text content specifically marked on the classroom blackboard is recognized as the first text information by means of OCR recognition.

Further, in a preferred embodiment provided by this application, the first text information includes at least one of teaching link information and knowledge point information.

The teaching link information here can be understood as the link corresponding to the recorded current image among the images in the sub-segments of the video stream that meet the preset duration. For example, the specific title currently also corresponds to the PPT. The knowledge point information here can be understood as the specific annotation content of the current image recorded in the image that meets the preset duration in the sub-segment of the video stream.

It is understandable that there will inevitably be some information unrelated to the teaching content in the relevant blackboard writing materials in the course of classroom teaching. This is called worthless information here. The identification of these worthless information will increase the proportion of meaningless content in the first text while increasing the amount of calculation. Therefore, by identifying the teaching link information or knowledge point information in the sub-segments of the video stream in a targeted manner, it is convenient to obtain more accurate first text identification information, reduce the redundancy of the first text information, and facilitate the multimedia analysis and summary. Sure.

Further, in a preferred embodiment provided by this application, the second text information specifically includes at least one of text error correction information, keyword information, question information, and emotion description information.

It can be understood that the second text information corresponds to the text information generated according to the sub-segments of the audio stream. In a classroom teaching scenario, the audio stream sub-segments are files recorded with sound information related to the classroom teaching process. In practical applications, identifying the sub-segment of the audio stream as the second text can be used to produce subtitle information synchronized with the sub-segment of the video stream. Through the natural language processing model, the text information in the subtitles can be corrected, and the keyword information, question information, and emotional description information can also be extracted. These information can be understood as the forming elements of the multimedia stream analysis summary. Therefore, the second text information obtained by analysis specifically includes at least one of text error correction information, keyword information, question information, and emotion description information. In this way, the accuracy of the multimedia stream analysis summary can be guaranteed.

Further, in a preferred implementation manner provided by this application, processing the scene information, the first text information and the second text information to form an analysis summary of the multimedia stream, specifically including: analyzing the scene information , the first text information and the second text information are cross-validated to form an analysis summary of the multimedia stream.

The cross-validation described here can be understood as summarizing various types of data such as scene information, first text information, and second text information obtained by analysis into different course structures. It can be understood that the analysis of the scene information and the first text information is performed according to the sub-segments of the video stream. The identification of the second text information is completed according to the sub-segments of the audio stream. The video stream sub-segments and the audio stream sub-segments are both extracted from the multimedia stream segments. If the analysis summary of the multimedia stream is performed only according to some information in the scene information, the first text information and the second text information, a multimedia stream analysis summary with high accuracy cannot be obtained. Therefore, it is necessary to comprehensively consider the scene information, the first text information and the second text information, and classify and summarize them according to the course structure, so as to obtain a high-accuracy multimedia stream analysis summary. This process can also be understood as completing the dismantling of the video content data according to the acquired multimedia video content, and reclassifying the dismantling data and relating it to the corresponding classroom link.

In a specific embodiment provided in this application, the course content in the classroom teaching scenario can be disassembled into three categories: teaching content, teacher-student behavior, and teacher-student language. Among them, the teaching content can be mainly reflected in the teaching PPT and the voice content of the teacher's lecture; the behavior of teachers and students is mainly reflected in the behavior changes of the movements and limbs; the language of teachers and students is mainly reflected in the language communication. Therefore, for the course content, the OCR recognition technology is used to identify the text content of the specific mark on the PPT/blackboard. At this time, the first division of the overall structure of the course is completed. Classroom sessions will be differentiated. Through face recognition technology combined with the recognition of action behaviors, a series of behaviors such as teacher teaching, students writing on the blackboard, raising their hands, standing up, reading, and writing can be divided into behaviors. Finally, the speech in the classroom is translated into real-time subtitles, and through the natural language processing ability, the text information in the subtitles can be corrected, and the language related to keyword information, question information, emotional description information, etc. can also be extracted. environment information. In this way, the related scene information, the first text information and the second text information are obtained. By summarizing the obtained scene information, first text information and second text information and other data into different course structures, the disassembly of the classroom teaching video content can be completed, and the relevant data obtained from the disassembly can be re-organized. A class is associated with a link to the corresponding class.

Embodiments of the present application further provide a multimedia stream processing apparatus 100, which is mainly used for processing multimedia files in complex scenarios. In a specific implementation manner provided by this application, the multimedia stream processing apparatus 100 may be used to process a multimedia file that records a complex scene of a classroom teaching process. Specifically, please refer to FIG. 2, a multimedia stream processing device, comprising:

an acquisition module 11, for acquiring multimedia stream segments;

The decoding module 12 is used to decode and obtain the sub-segment of the video stream and the sub-segment of the audio stream;

a video analysis module 13, configured to analyze the sub-segments of the video stream to generate scene information and first text information;

an audio analysis module 14, configured to analyze the sub-segments of the audio stream to generate second text information;

The analysis summary generating module 15 is configured to process the scene information, the first text information and the second text information to form an analysis summary of the multimedia stream.

The acquiring module 11 is used for acquiring multimedia stream segments. The multimedia stream segment here can be understood as a file that records media information such as text, graphics, video, animation, sound, and video of the corresponding scene. In a specific implementation manner provided by the present application, the acquired multimedia stream segment is a multimedia file with a certain duration and recorded with a classroom teaching scene. The multimedia stream segment can be obtained by shooting with a corresponding video shooting device. In this way, a real-time classroom scene can be photographed, thereby obtaining a multimedia file recording information such as voice, text, pictures, personnel objects, etc. in the course of classroom teaching.

The decoding module 12 is used to decode and obtain the sub-segments of the video stream and the sub-segments of the audio stream. The video stream sub-segment here can be understood as the image information in the multimedia segment. The audio stream sub-segment here can be understood as the sound information in the multimedia segment. Decode the acquired media stream segment, that is, extract the image information and sound information in the multimedia file with a certain duration, and convert it into several consecutive frames of images and continuous audio in the preset file format, so as to obtain the corresponding multimedia stream segment. Video stream subclip and audio stream subclip.

When the acquired multimedia stream segment is a multimedia file with a certain duration and recorded with classroom teaching scenes, after decoding, correspondingly obtained video stream sub-segments that record information such as words, pictures, personnel objects in the classroom teaching process, and record with Audio stream sub-segments of sound information during classroom teaching.

The video analysis module 13 is configured to analyze the sub-segments of the video stream to generate scene information and first text information. It can be understood that several consecutive frames of images in a multimedia stream segment can constitute a video stream sub-segment. Each frame of image is recorded with corresponding scene information. In a classroom teaching scenario, the video stream sub-segments are several consecutive frames of images that record information such as words, pictures, personnel objects, etc. in the course of classroom teaching. After the analysis of the AI module with corresponding functions, the specific scene information corresponding to the human object in the sub-segment of the current video stream, and the specific text information corresponding to the words and pictures in the sub-segment of the current video stream can be obtained.

Specifically, by identifying the human object in the sub-segment of the current video stream, the specific action category of the current human object can be determined, thereby facilitating the determination of the specific classroom teaching scene corresponding to the sub-segment of the current video stream. By identifying the specific text information corresponding to the text and picture in the sub-segment of the current video stream, the specific text type or description content corresponding to the text and picture information corresponding to the sub-segment of the current video stream can be determined, so that the corresponding sub-segment of the current video stream can be generated. the first text message. The first text information here can be understood as text information generated according to the sub-segments of the video stream.

The audio analysis module 14 is configured to analyze the sub-segments of the audio stream to generate second text information. It can be understood that the audio stream sub-segment can be generated according to the voice information in the multimedia stream segment. In a classroom teaching scenario, the audio stream sub-segments are files that record sound information related to the classroom teaching process. The AI module with corresponding functions can determine the specific narration content of the audio stream sub-segment, and obtain second text information corresponding to the narration content of the audio stream sub-segment. The second text information here can be understood as text information generated according to the sub-segments of the audio stream.

The analysis summary generating module 15 is configured to process the scene information, the first text information and the second text information to form an analysis summary of the multimedia stream. The analysis summary here can be understood as an overview of the specific real-time scene corresponding to the currently processed multimedia stream segment. The processing of the scene information, the first text information and the second text information is mainly to identify the key data with a high degree of correlation between the real-time scene corresponding to the multimedia stream segment. By integrating the identified target data, the specific teaching process corresponding to the currently processed multimedia stream segment can be obtained. Splitting multimedia stream segments into video stream sub-segments and audio stream sub-segments, and analyzing them through AI modules with corresponding functions, can effectively reduce the amount of data processed by an AI module with a single function, and can accurately select the ones with corresponding analysis functions. Analysis module, thereby improving the identification efficiency of multimedia stream segments.

Further, in a preferred embodiment provided by this application, the video analysis module 13 is used to analyze the sub-segments of the video stream to generate scene information, and is specifically used to: analyze the sub-segments of the video stream to generate object-oriented identity features Identification information and description information of the object's action behavior.

Here, the identity feature identification information of the object can be understood as the facial feature information of the object. It can be understood that the specific identity information of the person object can be determined by acquiring an image with the face of a certain person object and recognizing the facial features of the person object through a pre-trained recognition algorithm. For example, determine information such as the name and student ID of a certain student, or information such as the name and job ID of a certain teacher.

The description information of the action behavior of the object here can be understood as the specific action category of the person object in the sub-segment of the current video stream. It can be understood that, by acquiring an image with a body movement of a certain human object, and recognizing the image by a pre-trained recognition algorithm, the specific action category of the human object can be determined. For example, it is determined that the current action behavior of the person object is writing behavior, standing up behavior, or writing on the blackboard.

By identifying the specific identity information and behavior information of the objects involved in the sub-segments of the video stream, the behavior of teachers and students in the teaching scene corresponding to the sub-segments of the video stream can be determined, thereby improving the accuracy of the multimedia stream analysis summary.

Further, in a preferred embodiment provided by this application, the video analysis module 13 is configured to analyze the sub-segments of the video stream to generate the first text information, and is specifically used to: analyze the sub-segments of the video stream to generate the action behavior of the object The first text message pointed to.

The first text information pointed to by the action behavior of the object here can be understood as text information having a certain degree of correlation with the specific behavior of the person object. It can be understood that the first text information generated according to the sub-segments of the video stream includes specific text types or description contents corresponding to the text and picture information in the classroom teaching scenario. For example, the PPT display page in the classroom, the blackboard report information related to the classroom background. In the classroom teaching scenario, the text information in the PPT display page in the classroom is the text information related to the action behavior of the person and object. However, the information reported on the blackboard is background information in the sub-segment of the video stream, and has nothing to do with the actions and behaviors of people in the current scene, so it does not belong to the first text information.

By analyzing the first text information related to the actions of people and objects in the sub-segments of the video stream, the data processing amount of the corresponding functional modules can be reduced, and the recognition accuracy of the corresponding functional modules can be increased, thereby effectively improving the first text information. analytical efficiency.

Further, in a preferred embodiment provided by this application, the video analysis module 13 is used to analyze the sub-segments of the video stream, and generate the first text information pointed to by the action behavior of the object, which is specifically used for: analyzing the sub-segments of the video stream, Acquire an image that lasts for a preset duration; use OCR to identify the image to generate first text information.

It can be understood that the first text information that the object action behavior points to is text information that has a certain degree of correlation with the object behavior in the scene corresponding to the current video stream sub-segment. In the classroom teaching scenario, the text information that has a certain degree of relevance with the behavior of the object can be understood as the text information related to the teaching content. For example, the blackboard written by the teacher, the text information in the PPT display page, etc. It should be pointed out that, in the actual teaching process, if the text information pointed to by the object's action behavior is more important, the human object will continue for a certain period of time when the relevant behavior of the corresponding text information is carried out. That is, images with important textual information stay longer. When the text information pointed to by the action behavior of the object is less important or worthless, it has a correspondingly shorter duration. That is, images with unimportant text information have shorter dwell time. Therefore, the importance of the text in the image can be judged according to the duration of the image. The image duration here can be preset according to actual conditions or empirical values. If the duration of a certain frame of image satisfies the preset condition, the subsequent text recognition process can be started. Correspondingly, if the duration of a certain frame of image does not meet the preset condition, it means that the corresponding text is of low importance, and corresponding identification does not need to be carried out. In this way, the recognition efficiency of the first text can be increased on the basis of ensuring the recognition accuracy of the first text.

Specifically, when recognizing the image memorization text information that satisfies the preset duration, the OCR recognition method can be used. That is, the optical character recognition technology is used to recognize the character information in the images satisfying the preset conditions in the sub-segments of the video stream. For example, the relevant content in the PPT page that satisfies the preset condition in the classroom display PPT is identified as the first text information by the OCR identification method. Alternatively, the text content specifically marked on the classroom blackboard is recognized as the first text information by means of OCR recognition.

Further, in a preferred embodiment provided by this application, the first text information includes at least one of teaching link information and knowledge point information.

The teaching link information here can be understood as the link corresponding to the recorded current image among the images in the sub-segments of the video stream that meet the preset duration. For example, the specific title currently also corresponds to the PPT. The knowledge point information here can be understood as the specific annotation content of the current image recorded in the image that meets the preset duration in the sub-segment of the video stream.

It is understandable that there will inevitably be some information unrelated to the teaching content in the relevant blackboard writing materials in the course of classroom teaching. This is called worthless information here. The identification of these worthless information will increase the proportion of meaningless content in the first text while increasing the amount of calculation. Therefore, by identifying the teaching link information or knowledge point information in the sub-segments of the video stream in a targeted manner, it is convenient to obtain more accurate first text identification information, reduce the redundancy of the first text information, and facilitate the multimedia analysis and summary. Sure.

Further, in a preferred embodiment provided by this application, the second text information specifically includes at least one of text error correction information, keyword information, question information, and emotion description information.

It can be understood that the second text information corresponds to the text information generated according to the sub-segments of the audio stream. In a classroom teaching scenario, the audio stream sub-segments are files recorded with sound information related to the classroom teaching process. In practical applications, identifying the sub-segment of the audio stream as the second text can be used to produce subtitle information synchronized with the sub-segment of the video stream. Through the natural language processing model, the text information in the subtitles can be corrected, and the keyword information, question information, and emotional description information can also be extracted. These information can be understood as the forming elements of the multimedia stream analysis summary. Therefore, the second text information obtained by analysis specifically includes at least one of text error correction information, keyword information, question information, and emotion description information. In this way, the accuracy of the multimedia stream analysis summary can be guaranteed.

Further, in a preferred embodiment provided by the present application, the analysis summary generation module 15 is configured to process the scene information, the first text information and the second text information to form a summary of the multimedia stream. The analysis summary is specifically used for: cross-validating the scene information, the first text information and the second text information to form the analysis summary of the multimedia stream.

The cross-validation described here can be understood as summarizing various types of data such as scene information, first text information, and second text information obtained by analysis into different course structures. It can be understood that the analysis of the scene information and the first text information is performed according to the sub-segments of the video stream. The identification of the second text information is completed according to the sub-segments of the audio stream. The video stream sub-segments and the audio stream sub-segments are both extracted from the multimedia stream segments. If the analysis summary of the multimedia stream is performed only according to some information in the scene information, the first text information and the second text information, a multimedia stream analysis summary with high accuracy cannot be obtained. Therefore, it is necessary to comprehensively consider the scene information, the first text information and the second text information, and classify and summarize them according to the course structure, so as to obtain a high-accuracy multimedia stream analysis summary. This process can also be understood as completing the dismantling of the video content data according to the acquired multimedia video content, and reclassifying the dismantling data and relating it to the corresponding classroom link.

It should be noted that the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a series of elements includes not only those elements, but also no Other elements expressly listed, or which are also inherent to such a process, method, article of manufacture or apparatus. Without further limitation, the phrase "comprising a..." qualifying an element does not preclude the presence of additional identical elements in a process, method, article of manufacture, or apparatus that includes the element.

The above descriptions are merely examples of the present application, and are not intended to limit the present application. Various modifications and variations of this application are possible for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.

Claims (3)

1. A method for processing a multimedia stream, comprising the steps of:

acquiring a multimedia stream fragment;

decoding to obtain a video stream sub-segment and an audio stream sub-segment;

analyzing the video stream sub-segment to generate object-oriented identity characteristic identification information, description information of object action behaviors and first text information pointed by the object action behaviors;

analyzing the audio stream sub-segment to generate second text information, wherein the second text information at least specifically comprises one of text error correction information, keyword information, question information and emotion description information;

carrying out cross validation on object-oriented identity characteristic identification information, description information of object action behaviors, first text information and second text information to form an analysis abstract of the multimedia stream;

the first text information at least comprises one of teaching link information and knowledge point information.

2. The method for processing multimedia stream according to claim 1, wherein analyzing the video stream sub-segment to generate the first text information pointed by the object action behavior, specifically comprises:

analyzing the video stream sub-segments to obtain images lasting for a preset time;

and recognizing the image by using OCR to generate first text information.

3. A multimedia stream processing apparatus, comprising:

the acquisition module is used for acquiring the multimedia stream fragments;

the decoding module is used for decoding and acquiring the video stream sub-segment and the audio stream sub-segment;

the video analysis module is used for analyzing the video stream sub-segments to generate object-oriented identity feature identification information, description information of object action behaviors and first text information pointed by the object action behaviors;

the audio analysis module is used for analyzing the audio stream sub-segments to generate second text information, wherein the second text information at least specifically comprises one of text error correction information, keyword information, question information and emotion description information;

the analysis abstract generating module is used for performing cross validation on the object-oriented identity characteristic identification information, the description information of the object action behavior, the first text information and the second text information to form an analysis abstract of the multimedia stream;

the first text information at least comprises one of teaching link information and knowledge point information.

Images