JP7807460B2 - Improved virtual meeting user experience based on augmented intelligence - Google Patents

Description

The present invention relates generally to the field of data processing, and more particularly to a system and method based on augmented intelligence for improving virtual meeting user experience.

Augmented intelligence is an alternative conceptualization of artificial intelligence (AI) that emphasizes the supporting role of AI in enhancing human capabilities. Augmented intelligence uses machine learning and deep learning to enhance, rather than replace, human intelligence. Augmented intelligence enhances human intelligence by improving human decision-making, which in turn allows actions to be taken in response to the improved decisions.

Machine learning describes AI's ability to learn and improve from experience without additional programming. Natural language processing, which enables computers to identify human language, is an example of machine learning. Deep learning describes AI processes that mimic the human brain's ability to process data and examine patterns.

A key difference between augmented intelligence and AI is the difference in autonomy. Augmented intelligence processes large amounts of data that would otherwise overwhelm a human decision maker, eliminating factors such as bias, fatigue, or distraction that would cause the data to be distorted or misinterpreted. Augmented intelligence analyzes the data, identifies patterns, and reports those patterns to the user, after which human intelligence can take over. Augmented intelligence enables the system to learn from the user about their individual preferences and expectations, providing a personalized experience tailored to their preferences.

An example of augmented intelligence is viewing recommendations provided by a streaming video service. Augmented intelligence algorithms analyze a user's viewing habits and recommend additional viewings based on those habits. The user is then left to decide whether to act on the recommendations. Augmented intelligence also has applications in mining big data for patterns and predictive indicators in any industry. Examples include, but are not limited to, online stores using data analytics to predict customer preferences, virtual customer service assistants based on natural language processing, political think tanks using big data analytics to identify swing voters, medical analysis of case files to identify efficient treatment options, factory automation overseen by human employees, predictive maintenance of factory equipment based on historical data, mobile video games that use the surrounding environment and data to create augmented reality events and overlay computer-generated images on a smartphone camera screen, autopilot systems for airplanes and drones, and investment and finance applications that monitor and identify stock market patterns.

AI, on the other hand, operates without any human assistance. Examples of AI are email spam filters or AI-powered search suggestions.

Aspects of an embodiment of the present invention disclose a method, computer program product, and computer system for improving a virtual conference user experience. A processor detects a user who has been absent from a virtual conference including at least two or more participants for a preset period of time or a preset percentage of the pre-scheduled virtual conference's allotted total time. The processor retrieves from a database a first set of data regarding the user, a second set of data regarding at least two participants of the virtual conference, and a third set of data regarding relationships between the user and the at least two participants of the virtual conference. The processor prepares a summary tailored to the user's profile and covering the portion of the virtual conference during which the user was absent. The processor detects that the user reconnects to the virtual conference. The processor determines whether the user will review the summary before rejoining the virtual conference based on preset user preferences, a user-made decision, or a machine-driven recommendation. In response to determining that the user will review the summary before rejoining the virtual conference, the processor prompts the user to review the summary using a default set of user preferences. The processor outputs the summary to the user.

In some aspects of embodiments of the present invention, after outputting the summary to the user, the processor compares the summary to a playback of the full recording of the virtual meeting. The processor requests feedback from the user. The processor receives feedback from the user. The processor uses reinforcement learning to improve the accuracy of multiple future summaries that are prepared. The processor stores the user feedback in a database.

In some aspects of embodiments of the present invention, a processor identifies two or more participants in a virtual conference. The processor assigns weights to the two or more participants in the virtual conference based on a first set of factors. The processor ranks the two or more participants in the virtual conference based on the weights assigned to the two or more participants.

In some aspects of embodiments of the present invention, a processor ranks at least two participants of the virtual conference based on weights assigned to the at least two participants, and then extracts a plurality of audio frames, a plurality of video frames, and a plurality of audio and video frames from a portion of the virtual conference during which the users were disconnected. The processor identifies context for the plurality of audio frames, the plurality of video frames, and the plurality of audio and video frames to understand one or more topics of the virtual conference.

In some aspects of embodiments of the present invention, after identifying the context of the plurality of audio frames, the plurality of video frames, and the plurality of audio and video frames to understand one or more topics of the virtual conference, the processor selects a subset of the plurality of audio frames, a subset of the plurality of video frames, and a subset of the plurality of audio and video frames based on whether the plurality of audio frames, the plurality of video frames, and the plurality of audio and video frames are relevant to a user's profile. The processor ranks the subset of the plurality of audio frames, the subset of the plurality of video frames, and the subset of the plurality of audio and video frames using an algorithmic method or a second set of factors. The processor merges the subset of the plurality of audio frames, the subset of the plurality of video frames, and the subset of the plurality of audio and video frames ranked above an algorithmically determined threshold. The processor prepares a unified summary of a subset of the plurality of audio frames ranked above an algorithmically determined threshold, a unified summary of a subset of the plurality of video frames ranked above an algorithmically determined threshold, and a unified summary of a subset of the plurality of audio and video frames ranked above an algorithmically determined threshold.

In some aspects of embodiments of the present invention, the processor prepares an integrated summary of a subset of the plurality of audio frames ranked above an algorithmically determined threshold, an integrated summary of a subset of the plurality of video frames ranked above an algorithmically determined threshold, and an integrated summary of a subset of the plurality of audio and video frames ranked above an algorithmically determined threshold, and then converts the integrated summary into one or more sentences using an extractive text summarization algorithm. The processor applies speaker diarization. The processor ranks the one or more sentences based on a third set of factors. The processor retains a subset of the one or more sentences ranked above a second threshold based on a second preset user preference or a second machine-driven recommendation.

In some aspects of embodiments of the present invention, the processor identifies one or more keywords within the subset of one or more sentences after retaining the subset of one or more sentences ranked above a second threshold based on a second preset user preference or a second machine-driven recommendation. The processor assigns weights to one or more keywords within the subset of one or more sentences based on the relevance of the one or more keywords to one or more topics of the virtual conference and based on the relevance of the one or more keywords to at least two participants of the virtual conference. The processor tags the subset of one or more sentences with the one or more keywords.

In some aspects of embodiments of the present invention, the processor tags the one or more subsets of sentences with one or more keywords and then prepares a global ranking incorporating at least two participants, the one or more keywords, and the one or more subsets of sentences. The processor prepares a summary tailored to the user's profile based on the global ranking.

In some aspects of embodiments of the present invention, the first set of data regarding the user, the second set of data regarding at least two participants in the virtual conference, and the third set of data regarding relationships between the user and the at least two participants in the virtual conference collected from the database include a profile created by the user, a company profile for the user, the user's calendar, one or more profiles created by the at least two participants in the virtual conference, company profiles for the at least two participants in the virtual conference, calendars for the at least two participants in the virtual conference, one or more presentation tools used by the at least two participants in the virtual conference, and data from one or more previous virtual conferences hosted, participated in, or attended by the user.

In some aspects of embodiments of the present invention, the processor captures the time during the virtual conference when the user reconnects. The processor calculates the total duration of time the user was disconnected from the virtual conference.

In some aspects of embodiments of the present invention, the set of default user preferences for reviewing summaries includes language preferences, viewing mode preferences, audio preferences, and subtitle preferences.

In some aspects of embodiments of the present invention, a processor extracts a first set of one or more key entities from the summary and a second set of one or more key entities from the full recording of the virtual conference. The processor matches the first set of one or more key entities from the summary with the second set of one or more key entities from the full recording of the virtual conference.

In some aspects of embodiments of the present invention, the processor provides the user with three or more choices representing the user's satisfaction level, the three or more choices including dissatisfied, neutral, and satisfied.

In some aspects of embodiments of the present invention, the first set of factors includes the roles of the at least two participants in the virtual conference, the roles of the at least two participants in the enterprise, and the association of the at least two participants with a user remote from the virtual conference.

In some aspects of embodiments of the present invention, the algorithmically determined threshold is dynamic and tailored to the training data set.

In some aspects of embodiments of the present invention, the second set of factors includes whether the frame includes the name of a user who has disconnected from the virtual conference, a ranking of at least two participants, and the association of at least two participants with the user who has disconnected from the virtual conference based on the user's technology interests, the user's expertise, or past analysis.

In some aspects of embodiments of the present invention, a processor detects changes between at least two participants in a plurality of frames of audio. The processor groups segments of audio together based on characteristics of the at least two participants.

In some aspects of embodiments of the present invention, the third set of factors includes the roles of at least two participants in the virtual meeting and the relevance of the one or more statements to one or more topics of the virtual meeting.

1 is a functional block diagram illustrating a distributed data processing environment in accordance with an embodiment of the present invention. 2 is a flowchart illustrating the operational steps of a configuration component of a user-specific abstraction program in a distributed data processing environment, such as the distributed data processing environment illustrated in FIG. 1, in accordance with an embodiment of the present invention. 2 is a flowchart illustrating the operational steps of a user-specific summary program in a distributed data processing environment, such as the distributed data processing environment shown in FIG. 1, in accordance with an embodiment of the present invention. 2 is a flowchart illustrating the operational steps of a machine learning component of a user-specific summarization program in a distributed data processing environment, such as the distributed data processing environment shown in FIG. 1, according to an embodiment of the present invention. 2 is a block diagram of components of a computing device in a distributed data processing environment, such as the distributed data processing environment shown in FIG. 1, according to an embodiment of the present invention.

Embodiments of the present invention recognize that problems may occur during a conference on a virtual collaboration server tool. For example, technical issues may cause a user to be disconnected from the conference. Due to this disconnection, the user may miss important information, such as a host's or another participant's information about an issue or topic, a host's response to another participant's question or concern, a progress report of actions taken by a participant, or expected takeaways and action items directed to the user who was disconnected from the conference.

Embodiments of the present invention provide an augmented intelligence-based system and method for improving the virtual conference user experience. Embodiments of the present invention propose preparing a summary tailored to the profile of a user who is away from a conference on a virtual collaboration server tool, covering the portion of the conference during which the user was disconnected.

Embodiments of the present invention detect when a user leaves a conference, either explicitly or implicitly. After detecting the user's departure, embodiments of the present invention apply the concepts of speech-to-text summarization using (1) merging and (2) extracted text summarization algorithms to the audio and/or video frames extracted from the portion of the conference during which the user was disconnected.

Embodiments of the present invention use speaker diarization to divide sentences into homogeneous segments according to participant identification information and identify keywords from the participants' sentences. Embodiments of the present invention assign weights to participants and keywords. Embodiments of the present invention prepare a global ranking that incorporates the identified and weighted participants, the identified and weighted keywords, and the ranked sentences.

Embodiments of the present invention reduce sentences to short summaries in the form of either a 30-second video or a page of text. Embodiments of the present invention tailor summaries to the profile of users away from the conference on the virtual collaboration server tool using factors such as each user's company profile, the user's company hierarchy, the user's position within the company hierarchy, the user's seniority within the user's company, the user's primary job, the user's primary job function, the user's technology interests, the user's expertise, and the user's history of involvement in conferences the user has hosted, participated in, or attended. Embodiments of the present invention also consider calculated speech recognition confidence scores and intent confidence scores as part of the summarization process.

Embodiments of the present invention leverage the weight of a user's relationships with other participants to determine when to prompt the user to use the summary after the user reconnects to the conference. Embodiments of the present invention provide the user with the option to play the summary during or after the conference. Embodiments of the present invention also provide the user with the option to play a summary that includes audio, video, subtitles, or any combination of these three formats.

Embodiments of the present invention collect feedback from users to improve future preparation of summaries.

Implementations of embodiments of the present invention may take a variety of forms, and exemplary implementation details are described below with reference to the figures.

FIG. 1 is a functional block diagram illustrating a distributed data processing environment (generally designated 100) in accordance with one embodiment of the present invention. In the illustrated embodiment, distributed data processing environment 100 includes servers 120 and user computing devices 130 interconnected via network 110. Distributed data processing environment 100 may include additional servers, computers, computing devices, IoT sensors, and other devices not shown. FIG. 1 provides merely an example of one embodiment of the present invention and is not intended to imply any limitations with regard to the environments in which various embodiments may be implemented. Many modifications to the illustrated environment may be made by those skilled in the art without departing from the scope of the present invention, as set forth in the claims.

Network 110 operates as a computing network, which may be, for example, a telecommunications network, a local area network, a wide area network such as the Internet, or a combination of the three, and may include wired, wireless, or fiber optic connections. Network 110 may include one or more wired and/or wireless networks capable of receiving and transmitting data, audio, and/or video signals, including multimedia signals containing data, audio, and video information. In general, network 110 may be any combination of connections and protocols supporting communication between server 120, user computing devices 130, and other computing devices (not shown) in distributed data processing environment 100.

The server 120 operates to execute a user-specific summarization program 122 and transmit and/or store data in a database 126. In an embodiment, the server 120 may transmit data from the database 126 to a user computing device 130. In an embodiment, the server 120 may receive data in the database 126 from a user computing device 130. In one or more embodiments, the server 120 may be a standalone computing device, an administrative server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, transmitting, and processing data. In one or more embodiments, the server 120 may be a computing system utilizing clustered computers and components (e.g., database server computers, application server computers, etc.) that function as a single pool of seamless resources when accessed within a distributed data processing environment 100, such as within a cloud computing environment. In one or more embodiments, server 120 may be a laptop computer, tablet computer, netbook computer, personal computer, desktop computer, personal digital assistant, smartphone, or any programmable electronic device capable of communicating with user computing devices 130 and other computing devices (not shown) in distributed data processing environment 100 via network 110. Server 120 may include internal and external hardware components, as shown and described in more detail in FIG. 5.

The user-specific summary program 122 operates to prepare a summary tailored to the profile of a user who is away from a conference on the virtual collaboration server tool, covering the portion of the conference while the user was disconnected.

In the illustrated embodiment, the user-specific summarization program 122 includes a machine learning component 124. In the illustrated embodiment, the user-specific summarization program 122 is a standalone program. In another embodiment, the user-specific summarization program 122 may be integrated into another software product, such as a virtual conferencing software package. In the illustrated embodiment, the user-specific summarization program 122 resides on the server 120. In other embodiments, the user-specific summarization program 122 may reside on the user computing device 130 or another computing device (not shown), provided that the user-specific summarization program 122 has access to the network 110.

In an embodiment, a user opts in to the user-specific summarization program 122 and configures a user profile using the user-specific summarization program 122. The configuration components of the user-specific summarization program 122 are shown and described in further detail with respect to FIG. 2. The operational steps of the user-specific summarization program 122 are shown and described in further detail with respect to FIG. 3. The operational steps of the machine learning component 124 of the user-specific summarization program 122 are shown and described in further detail with respect to FIG. 4.

The database 126 acts as a repository of data received, used, and/or generated by the user-specific summarization program 122. The database is a structured collection of data. The data includes, but is not limited to, a plurality of user profiles including information entered by the user during setup regarding each user's company profile, the user's company hierarchy, the user's position within the company hierarchy, the user's seniority within the user's company, the user's primary job, the user's primary job function, the user's technology interests, and the user's expertise; data from a current meeting on the virtual collaboration server tool and/or data from previous meetings on the virtual collaboration server tool that the user hosted, participated in, or attended; user preferences; alert notification preferences; and any other data received, used, and/or generated by the user-specific summarization program 122.

Database 126 may be implemented using any type of device capable of storing data and configuration files that can be accessed and utilized by server 120, such as a hard disk drive, database server, or flash memory. In an embodiment, database 126 is accessed by user-specific summary program 122 to store data, access data, or both. In the illustrated embodiment, database 126 resides on server 120. In another embodiment, database 126 may reside on another computing device, server, cloud server, or be distributed across multiple devices (not shown) anywhere in distributed data processing environment 100, provided that user-specific summary program 122 has access to database 126.

The present invention may involve a variety of accessible data sources, such as database 126, which may contain personal and/or business-sensitive data, content, or information that a user desires not to be processed. Processing refers to any automated or non-automated action or set of actions, such as collecting, recording, organizing, structuring, storing, adapting, altering, retrieving, consulting, using, disclosing by transmitting, disseminating, or otherwise making available, combining, restricting, erasing, or destroying personal and/or business-sensitive data. The user-specific abstraction program 122 enables authorized and secure processing of personal data. All storage of data in database 126 must be performed in accordance with local law and any appropriate authorizations obtained.

The user-specific summary program 122 provides informed consent along with notice of the collection of personal data and/or confidential business data, allowing the user to opt in or out of processing the personal data and/or confidential business data. Consent can take multiple forms. Opt-in consent can force the user to take affirmative action before the personal data and/or confidential business data are processed. Alternatively, opt-out consent can force the user to take affirmative action to prevent the processing of the personal data and/or confidential business data before the personal data and/or confidential business data are processed. The user-specific summary program 122 provides information about the personal data and/or confidential business data and the nature of the processing (e.g., type, scope, purpose, duration, etc.). The user-specific summary program 122 provides the user with a copy of the stored personal data and/or confidential business data. The user-specific summarization program 122 allows for the correction or completion of incorrect or incomplete personal data and/or confidential business data. The user-specific summarization program 122 allows for the immediate deletion of personal data and/or confidential business data.

The user computing device 130 operates to execute the user interface 132 and is associated with a user. In an embodiment, the user computing device 130 may be an electronic device such as a laptop computer, a tablet computer, a netbook computer, a personal computer, a desktop computer, a smartphone, or any programmable electronic device capable of executing the user interface 132 and communicating (i.e., sending data to and receiving data from) the user-specific summary program 122 over the network 110. In the illustrated embodiment, the user computing device 130 includes an instance of the user interface 132. The user computing device 130 may include components as described in further detail in FIG. 5.

The user interface 132 acts as a local user interface between the user-specific summary program 122 on the server 120 and the user of the user computing device 130. In some embodiments, the user interface 132 is a graphical user interface (GUI), a web user interface (WUI), or a voice user interface (VUI), or a combination thereof, and can display (i.e., visually display) or present (i.e., audibly present) text, documents, web browser windows, user options, application interfaces, and instructions for operation sent to the user from the user-specific summary program 122 over the network 110. The user interface 132 can also display or present alert notifications containing information (e.g., graphics, text, and/or sound) sent to the user from the user-specific summary program 122 over the network 110. In an embodiment, the user interface 132 is capable of transmitting and receiving data (i.e., to and from, respectively, the user-specific summary program 122 over the network 110).

Through the user interface 132, users can opt in to the user-specific summary program 122, create a user profile, enter information about each user's company profile, the user's company hierarchy, the user's position within the company hierarchy, the user's seniority within the user's company, the user's primary job, the user's primary job function, the user's technology interests, and the user's expertise, and set user preferences and alert notification preferences.

User preferences are settings that can be customized for a particular user. A set of default user preferences is assigned to each user of the user-specific summarization program 122. A user preference editor can be used to update values to change the default user preferences. User preferences that can be customized include, but are not limited to, general user system settings, specific user profile settings for the user-specific summarization program 122, alert notification settings, and machine-learned data collection/storage settings. Machine-learned data includes, but is not limited to, past results of iterations of the user-specific summarization program 122 and data regarding the user's previous responses to alert notifications sent by the user-specific summarization program 122. The machine-learned data comes from the user-specific summarization program 122, which is tailored to the user's profile when away from a meeting on the virtual collaboration server tool, self-learns how to prepare a summary covering the portion of the meeting while the user was away, and whether to prompt the user to review the summary before rejoining the meeting. The user-specific summarization program 122 self-learns by tracking the user's activity and improves with each iteration of the user-specific summarization program 122.

2 is a flowchart generally designated 200 illustrating operational steps of a setup component of a user-specific summary program 122 in the distributed data processing environment 100 of FIG. 1 in accordance with an embodiment of the present invention. In an embodiment, the user-specific summary program 122 completes a one-time setup with a user who will host, participate in, or attend a conference on the virtual collaboration server tool. The one-time setup enables the user-specific summary program 122 to capture relevant information about the user to create a user profile. In an embodiment, the user-specific summary program 122 receives a request to opt in from the user. In an embodiment, the user-specific summary program 122 requests information from the user. In an embodiment, the user-specific summary program 122 receives the requested information from the user. In an embodiment, the user-specific summary program 122 creates a user profile. In an embodiment, the user-specific summary program 122 stores the user profile. It should be understood that the process illustrated in FIG. 2 represents one possible iteration of the user-specific summary program 122, and that this process may be repeated for each opt-in request received by the user-specific summary program 122.

At step 210, the user-specific summary program 122 receives a request to opt in from the user. In an embodiment, the user-specific summary program 122 receives a request from the user to opt in to the user-specific summary program 122. In an embodiment, the user-specific summary program 122 receives a request from the user to opt in to the user-specific summary program 122 via the user interface 132 of the user computing device 130. By opting in, the user agrees to share their data with the database 126.

In step 220, the user-specific summarization program 122 requests information from the user. In an embodiment, the user-specific summarization program 122 requests information from the user via the user interface 132 of the user computing device 130. The information requested from the user includes, but is not limited to, information about the user's preferences (e.g., general user system settings, such as alert notifications, for the user computing device 130), information about alert notification preferences (e.g., alert notifications are sent when the user reconnects to a conference they left or alert notifications are sent when a conference they left completes), and information necessary to create a user profile (e.g., information about each user's company profile, the user's company hierarchy, the user's position within the company hierarchy, the user's seniority within the user's company, the user's primary job, the user's primary job function, the user's technology interests, and the user's expertise). In an embodiment, in response to the user-specific summarization program 122 receiving a request to opt in from the user, the user-specific summarization program 122 requests information from the user.

In step 230, the user-specific summary program 122 receives the requested information from the user. In an embodiment, the user-specific summary program 122 receives the requested information from the user via the user interface 132 of the user computing device 130. In an embodiment, the user-specific summary program 122 receives the requested information from the user in response to the user-specific summary program 122 requesting information from the user.

In step 240, the user-specific summary program 122 creates a user profile. In an embodiment, the user-specific summary program 122 creates a user profile for the user. In an embodiment, the user-specific summary program 122 creates a user profile that includes the user's preferences and the user's alert notification preferences in addition to information entered by the user during setup for the user. In an embodiment, in response to the user-specific summary program 122 receiving the requested information from the user, the user-specific summary program 122 creates the user profile.

In step 250, the user-specific summary program 122 stores the user profile. In an embodiment, the user-specific summary program 122 stores the user profile in a database (e.g., database 126). In an embodiment, in response to the user-specific summary program 122 creating the user profile, the user-specific summary program 122 stores the user profile.

FIG. 3 is a flowchart generally designated 300 illustrating operational steps of the user-specific summarization program 122 in the distributed data processing environment 100 of FIG. 1 in accordance with an embodiment of the present invention. In an embodiment, the user-specific summarization program 122 operates to detect when a user disconnects from a conference on the virtual collaboration server tool, prepare a summary tailored to the user's profile while disconnected from the conference on the virtual collaboration server tool and covering the portion of the conference while the user was disconnected, provide the summary to the user when the user reconnects to the conference, and collect feedback from the user to assist the user-specific summarization program 122 in preparing more tailored summaries in future iterations. It should be understood that the process depicted in FIG. 3 illustrates one possible iteration of the process flow, and this process may be repeated for each conference the user hosts, participates in, or attends on the virtual collaboration server tool.

In step 305, the user-specific summarizing program 122 detects when the user leaves a conference on a virtual collaboration server tool (e.g., Cisco Webex®, Zoom®, Google Meet®, Microsoft® Teams, etc.). In an embodiment, the user-specific summarizing program 122 detects when the user leaves a conference on a virtual collaboration server tool in which one or more hosts and two or more participants are present. In an embodiment, the user-specific summarizing program 122 detects when the user leaves a conference on a virtual collaboration server tool in which the host and/or participants are utilizing one or more presentation tools (e.g., Microsoft® PowerPoint slides, Microsoft® Excel files, spreadsheets, web pages, diagrams, flowcharts, etc.).

In an embodiment, the user-specific summary program 122 detects explicit user departures initiated by the user (e.g., the user changes their status to Away, the user signs out of the meeting, etc.). In one or more embodiments, the user-specific summary program 122 detects implicit user departures caused by technical issues (e.g., network connection issues, device failure, power outage, etc.).

In a first example, User A, a computer science student, attends User A's online virtual class on a virtual collaboration server tool. One day, while attending class, User A experiences network connection problems. Due to the network connection problems experienced by User A, User A is disconnected from User A's online virtual class. The user-specific summary program 122 detects User A's implicit withdrawal from User A's online virtual class.

In a second example, User B, an employee of a technology company, works from home. User B frequently holds team meetings with User B's team on a virtual collaboration server tool. During one of User B's team meetings, User B disconnects from the team meeting to attend to a personal emergency. The user-specific summary program 122 detects User B's explicit withdrawal from User B's team meeting.

In a third example, User C, a taxpayer for City X, attends City X's online city council meetings on the virtual collaboration server tool. During one of the City X online city council meetings that User C is attending, User C disconnects from the meeting to take a break. The user-specific summary program 122 detects User C's explicit departure from City X's online city council meeting.

In an embodiment, the user-specific summary program 122 detects when a user has been away from a meeting on the virtual collaboration server tool for a preset period of time greater than 10 minutes but less than 30 minutes. In one or more embodiments, the user-specific summary program 122 detects when a user has been away from a meeting on the virtual collaboration server tool for a preset percentage of the pre-scheduled meeting's total allotted time (e.g., the user has been away for 10 to 25 percent (e.g., 6 to 15 minutes) of the pre-scheduled meeting's total allotted time (e.g., 60 minutes)).

In an embodiment, the user-specific summary program 122 captures the start time of a conference on the virtual collaboration server tool. In an embodiment, the user-specific summary program 122 captures the start time of the user's departure (i.e., the time during the conference when the user explicitly or implicitly disconnects).

In step 310, the user-specific summarization program 122 retrieves the data. In an embodiment, the user-specific summarization program 122 retrieves the data for the purpose of preparing a summary tailored to the profile of the user away from the conference on the virtual collaboration server tool, covering the portion of the conference while the user was disconnected.

In embodiments, the user-specific summarization program 122 retrieves data about the user. In one or more embodiments, the user-specific summarization program 122 retrieves data about other participants in the conference. In one or more embodiments, the user-specific summarization program 122 retrieves data about connections between the user and other participants in the conference. In one or more embodiments, the user-specific summarization program 122 retrieves data about interactions between participants, including specific calls of participants, questions asked by participants, and answers provided to questions asked by participants. In one or more embodiments, the user-specific summarization program 122 retrieves data about the topic of the conference.

In an embodiment, the user-specific summarization program 122 retrieves data from sources including, but not limited to, the user profile created in step 240, the user's company profile, the user's calendar, other participants' user profiles, other participants' company profiles, other participants' calendars, one or more presentation tools (e.g., Microsoft® PowerPoint slides, Microsoft® Excel files, spreadsheets, web pages, diagrams, flowcharts, etc.) used by the host and/or participants, and data stored in a database (e.g., database 126) from previous meetings hosted, participated in, or attended by the user. While examples of the user-specific summarization program 122 retrieving data are described herein using individual methods, it should be noted that the user-specific summarization program 122 may retrieve data via one or more combinations of the above embodiments. In an embodiment, the user-specific summarization program 122 retrieves data in response to the user-specific summarization program 122 detecting when the user has left the meeting on the virtual collaboration server tool.

In step 315, the user-specific summarization program 122 prepares a summary. In an embodiment, the user-specific summarization program 122 prepares a summary tailored to the profile of the user who is away from the conference on the virtual collaboration server tool, covering the portion of the conference while the user was disconnected. In an embodiment, the user-specific summarization program 122 prepares the summary by applying concepts of speech-to-text summarization using (1) merging and (2) extracting text summarization algorithms. In an embodiment, after the user-specific summarization program 122 detects the user's departure, an embodiment of the present invention prepares the summary by applying concepts of speech-to-text summarization using (1) merging and (2) extracting text summarization algorithms to audio frames and/or video frames extracted from the portion of the conference while the user was disconnected. In an embodiment, the user-specific summarization program 122 prepares the summary by applying the concepts of speech-to-text summarization using (1) merging and (2) extractive text summarization algorithms to one or more presentation tools (e.g., Microsoft® PowerPoint slides, Microsoft® Excel files, spreadsheets, web pages, diagrams, flowcharts, etc.) used by the host and/or participants.

Extractive text summarization algorithms are applied by identifying and extracting important sentences and other salient information from a particular audio, video, or text file, or a combination thereof. Weights are assigned to the sentences extracted from the text file, and the sentences are ranked based on their particular weight. Highly ranked sentences are grouped together to form a concise summary. Stated another way, extractive text summarization algorithms are applied by identifying and generating verbatim important sections of text, and generating a subset of sentences from the original text as a summary.

Step 315 is described in further detail with respect to flowchart 400 of FIG. 4. In an embodiment, in response to the user-specific summary program 122 retrieving the data, the user-specific summary program 122 prepares a summary.

In step 320, the user-specific summary program 122 detects when the user reconnects to the conference on the virtual collaboration server tool. In an embodiment, the user-specific summary program 122 captures the user's re-engagement time (i.e., the time during the conference when the user reconnects). In an embodiment, the user-specific summary program 122 calculates the total duration of the user's disengagement period (i.e., the total duration of time the user was disconnected from the conference). If the user-specific summary program 122 determines that the total duration of the user's disengagement period is longer than a preset period or a preset percentage of the pre-scheduled conference's total allotted time, the user-specific summary program 122 does not output a summary to the user. In an embodiment, in response to the user-specific summary program 122 preparing a summary, the user-specific summary program 122 detects when the user reconnects to the conference on the virtual collaboration server tool.

In a first example, User A reconnects to User A's online virtual class within 10 minutes of disconnection after resolving User A's network connection issues. The user-specific summary program 122 detects User A reconnecting, captures User A's re-engagement time, and calculates the total duration of User A's disengagement period from User A's online virtual class to be 10 minutes.

In a second example, User B reconnects to User B's team conference within 15 minutes of disconnection after attending to User B's personal emergency. The user-specific summary program 122 detects User B reconnecting, captures User B's re-engagement time, and calculates the total duration of User B's withdrawal period from User B's team conference to be 15 minutes.

In a third example, User C takes a break and then reconnects to City X's online city council within 10 minutes of disconnection. The user-specific summary program 122 detects User C reconnecting, captures User C's re-engagement time, and calculates the total duration of User C's withdrawal from City X's online city council to be 10 minutes.

At decision 325, the user-specific summarizing program 122 determines whether the user will review the summary before rejoining the conference. In embodiments, the user-specific summarizing program 122 determines whether the user will review the summary before rejoining the conference based on the user preferences set by the user in step 220. In one or more embodiments, the user-specific summarizing program 122 determines whether the user will review the summary before rejoining the conference based on a user decision. In one or more embodiments, the user-specific summarizing program 122 determines whether the user will review the summary before rejoining the conference based on a machine-driven recommendation. For example, the user-specific summarizing program 122 recommends that the user read the summary before rejoining because any information the user missed during the user's absence needs to be known before rejoining. Although examples of the user-specific summarizing program 122 determining whether the user will review the summary before rejoining the conference are described herein using individual methods, it should be noted that the user-specific summarizing program 122 may determine whether the user will review the summary before rejoining the conference via one or more combinations of the above embodiments. In an embodiment, in response to the user-specific summary program 122 detecting when the user reconnects to the conference on the virtual collaboration server tool, the user-specific summary program 122 determines whether the user should review the summary before rejoining the conference.

If the user-specific summarization program 122 determines that the user will review the summary before rejoining the conference (the "Yes" branch of decision 325), the user-specific summarization program 122 prompts the user to select how much the user prefers to review the summary (i.e., the user's preference) (step 330). If the user-specific summarization program 122 determines that the user will not review the summary before rejoining the conference (the "No" branch of decision 325), the user-specific summarization program 122 determines whether the conference has completed before prompting the user to select how much the user prefers to review the summary (decision 360).

At decision 360, the user-specific summary program 122 determines whether the conference is complete. In an embodiment, the user-specific summary program 122 determines that the conference is complete when the host ends the conference. In an embodiment, in response to the user-specific summary program 122 determining that the user will not review the summary before rejoining the conference, the user-specific summary program 122 determines whether the conference is complete.

If the user-specific summary program 122 determines that the conference is complete (the "Yes" branch of decision 360), the user-specific summary program 122 prompts the user to select how much the user prefers to review the summary (step 330). If the user-specific summary program 122 determines that the conference is not complete (the "No" branch of decision 360), the user-specific summary program 122 waits until the conference is complete before proceeding to step 330.

In the example, User C, a taxpayer for City X who is attending an online city council meeting for City X on the virtual collaboration server tool, reconnects to the online city council meeting for City X. The user-specific summarizing program 122 detects the reconnecting User C. Instead of delaying User C from rejoining the meeting, the user-specific summarizing program 122 makes a machine-driven recommendation that User C wait until the online city council meeting for City X is complete to review the summary. The user-specific summarizing program 122 does not prompt User C to select how much User C prefers to review the summary until the meeting is complete.

Returning to decision 325, if the user-specific summarization program 122 determines that the user will review the summary before rejoining the conference (the "Yes" branch of decision 325), the user-specific summarization program 122 proceeds to step 330 and prompts the user to select how the user prefers to review the summary. In step 330, the user-specific summarization program 122 prompts the user to select how the user prefers to review the summary. In an embodiment, the user-specific summarization program 122 prompts the user for default selections of settings, including, but not limited to, language, viewing mode, audio, and subtitles. The default selections provide the user with a recommended experience (e.g., English (American), high-bandwidth video viewing mode, audio on, and subtitles off) and are based on the user preferences set by the user in step 220. In an embodiment, the user-specific summarization program 122 prompts the user to change the default selections to select options that are more suitable for the user's work environment. For example, the user may change the default language setting to an alternate language (e.g., German, English (US), Spanish (Latin America), French, French (Canadian), Italian, Polish, Portuguese, Portuguese (Brazilian), etc.), change the default viewing mode to an alternate viewing mode (e.g., high bandwidth video, low bandwidth video, text, and graphics), change the default audio setting (e.g., on or off), or change the default subtitle setting (e.g., on or off), or any combination thereof. In an embodiment, user-specific summarization program 122 prompts the user, via user interface 132 of user computing device 130, to select how the user prefers to review the summary. In an embodiment, in response to user-specific summarization program 122 determining that the user will review the summary, user-specific summarization program 122 prompts the user to select how the user prefers to review the summary.

In step 335, the user-specific summary program 122 outputs the summary. In an embodiment, the user-specific summary program 122 outputs the summary in the format selected by the user in step 330. In an embodiment, the user-specific summary program 122 outputs the summary as an alert notification. In an embodiment, the user-specific summary program 122 outputs the summary to the user via the user interface 132 of the user computing device 130. In an embodiment, the user-specific summary program 122 outputs the summary in response to the user-specific summary program 122 prompting the user to select how the user prefers to review the summary.

In a first example, User A, a computer science student who was attending an online virtual class on a virtual collaboration server tool and experiencing network connection issues, reconnects to User A's online virtual class. The user-specific summary program 122 detects User A reconnecting and asks whether User A would prefer to review the summary before rejoining the class or to wait until the class is complete to review the summary. User A chooses to review the summary before rejoining User A's online virtual class. User A chooses to review the summary using the default selection and is taken to a breakout room. User A receives and views a 30-second video with audio. User A now has an update on what User A missed while disconnected and is ready to rejoin User A's online virtual class.

In a second example, User B, an employee of a technology company who works from home and was participating in a team conference with User B's team on a virtual collaboration server tool, reconnects to User B's team conference. The user-specific summarization program 122 detects User B reconnecting and asks if User B prefers to review the summary before rejoining the team conference or waits until the team conference is complete to review the summary. User B chooses to review the summary before rejoining the team conference. User B changes the default audio setting to "off" and the default subtitle setting to "on." User B remains in the current conference room and reviews the summary. User B receives and watches a 30-second video with subtitles. User B now has an update on what he missed while he was disconnected and is ready to rejoin User B's team conference.

In a third example, an online city meeting for City X is completed. The user-specific summary program 122 outputs a summary to User C, who receives and views a 30-second video with audio. User C now has an update on what he missed while he was disconnected from City X's online city meeting.

In step 340, the user-specific summarization program 122 compares the summary with a playback of the full recording of the meeting. In an embodiment, the user-specific summarization program 122 extracts key entities from the summary and from a playback of the full recording of the meeting. In an embodiment, the user-specific summarization program 122 matches similar key entities extracted from the summary and from a playback of the full recording of the meeting. In an embodiment, the user-specific summarization program 122 compares the summary with a playback of the full recording of the meeting to ensure that the meaning of the summary and the meeting are the same. In an embodiment, in response to the user-specific summarization program 122 outputting the summary, the user-specific summarization program 122 compares the summary with a playback of the full recording of the meeting.

In step 345, the user-specific summarization program 122 requests feedback from the user. In an embodiment, the user-specific summarization program 122 requests feedback from the user regarding the output of the summary in step 335. In an embodiment, the user-specific summarization program 122 provides the user with three options (i.e., -1, 0, +1) from which the user can select only one. The three options provided to the user represent the user's possible satisfaction levels (i.e., dissatisfied, neutral, and satisfied, respectively). In an embodiment, the user-specific summarization program 122 requests feedback from the user via the user interface 132 of the user computing device 130. In an embodiment, in response to comparing the summary with a playback of the full recording of the conference, the user-specific summarization program 122 requests feedback from the user.

In step 350, the user-specific summarization program 122 receives feedback from the user. In an embodiment, the user-specific summarization program 122 receives feedback from the user via the user interface 132 of the user computing device 130. In an embodiment, the user-specific summarization program 122 uses the feedback to improve the user-specific summarization program 122 in preparing more tailored summaries in future iterations of the process. In an embodiment, the user-specific summarization program 122 uses reinforcement learning to improve the user-specific summarization program 122. In an embodiment, the user-specific summarization program 122 uses ReLU or leaky ReLU as an activation function to improve accuracy. In an embodiment, the user-specific summarization program 122 performs speech-to-text (STT) recognition on the audio frames and/or video frames to prepare triplets. In an embodiment, the user-specific summarization program 122 recommends the top N ranked sentences to the end user. In an embodiment, the user-specific summarizer 122 uses a meeting summarizer as a reinforcement learning agent to model an iterative process after a Markov decision process during a collaboration process. In an embodiment, the user-specific summarizer 122 performs Q-learning techniques on state-action (SA) pairs to accelerate the execution time of the algorithm. The Q-learning technique serves as a crib sheet for the reinforcement learning (RL) agent. The Q-learning technique allows the RL agent to use feedback from the environment to learn the best actions it can perform in various environments. The Q-learning technique also uses a Q-value to track and improve the performance of the RL agent. Initially, the Q-value is set to an arbitrary value. However, as the RL agent performs various actions and receives feedback on the actions (i.e., dissatisfied, neutral, satisfied), the Q-value is updated. In an embodiment, the user-specific summarizer 122 designs the reinforcement learning system so that rewards increase over time. The maximum per-episode reward indicates that the RL agent has learned to perform the correct action by maximizing the total reward. In an embodiment, in response to the user-specific summary program 122 requesting feedback from the user, the user-specific summary program 122 receives feedback from the user.

In step 355, the user-specific summary program 122 stores the feedback. In an embodiment, the user-specific summary program 122 stores the feedback received from the user. In an embodiment, the user-specific summary program 122 stores the feedback received from the user in a database (e.g., database 126). In an embodiment, in response to the user-specific summary program 122 receiving the feedback from the user, the user-specific summary program 122 stores the feedback.

In some embodiments, the user-specific summary program 122 may perform steps 345, 350, and 355 as optional steps.

FIG. 4 is a flowchart generally designated 400 illustrating operational steps of the machine learning component 124 of the user-specific summarization program 122 in the distributed data processing environment 100 of FIG. 1 in accordance with an embodiment of the present invention. In an embodiment, the user-specific summarization program 122 operates to prepare summaries tailored to the profile of a user away from a conference on a virtual collaboration server tool, covering the portion of the conference while the user was disconnected, by applying concepts of merging, extractive summarization, speaker diarization, and machine learning. In an embodiment, the machine learning component 124 of the user-specific summarization program 122 runs continuously for the entire duration of a conference that the user hosts, participates in, or attends on the virtual collaboration server tool. It should be understood that the process illustrated in FIG. 4 illustrates one possible iteration of the process flow.

In step 405, the user-specific summarization program 122 identifies two or more participants of the conference. In an embodiment, the user-specific summarization program 122 identifies two or more participants of the conference via information collected from the virtual collaboration server tool (e.g., from a participant list or from an attendance report). In an embodiment, the user-specific summarization program 122 assigns weights to two or more participants of the conference. In an embodiment, the user-specific summarization program 122 assigns weights to two or more participants of the conference based on a set of factors. The set of factors includes, but is not limited to, the participant's role in the conference (i.e., the conference organizer, the requested attendee of the conference, the optional attendee of the conference, etc.), the participant's role in the participant's enterprise, and the participant's relevance to the user away from the conference. In an embodiment, the user-specific summarization program 122 ranks the two or more participants of the conference based on the participants' assigned weights (i.e., ranks the participants from most relevant to the user to least relevant to the user). In an embodiment, the user-specific summarization program 122 provides the user with the ability to override such rankings of two or more participants. In an embodiment, the user-specific summarization program 122 provides the user with the ability to override such rankings via the user interface 132 of the user computing device 130.

In step 410, the user-specific summary program 122 extracts multiple audio and/or video frames. In an embodiment, the user-specific summary program 122 extracts multiple audio and/or video frames from the portion of the conference from which the user was disconnected. In an embodiment, the user-specific summary program 122 continuously extracts multiple audio and/or video frames until a summary is prepared and output to the user who was disconnected from the conference.

In an embodiment, the user-specific summarization program 122 counts the number of audio and/or video frames extracted from the portion of the conference from which the user was disconnected. In an embodiment, the user-specific summarization program 122 counts the number of audio and/or video frames containing the user's name (e.g., audio and/or video frames during which the user's name was mentioned by the conference host because the user was expected to speak or answer a question, or audio and/or video frames during which the user's name was called by the conference host to join a smaller breakout group). In an embodiment, in response to the user-specific summarization program 122 identifying two or more participants in the conference, the user-specific summarization program 122 extracts multiple audio and/or video frames.

In step 415, the user-specific summarization program 122 identifies the context of the multiple audio frames and/or video frames. In an embodiment, the user-specific summarization program 122 identifies the context of the multiple audio frames and/or video frames extracted from a portion of the conference from which the user was detached. In an embodiment, the user-specific summarization program 122 identifies the context of the multiple audio frames and/or video frames to understand the intent and/or topic of the conference. In an embodiment, in response to the user-specific summarization program 122 extracting the multiple audio frames and/or video frames, the user-specific summarization program 122 identifies the context of the multiple audio frames and/or video frames.

In step 420, the user-specific summarization program 122 selects a subset of the plurality of audio frames and/or video frames. In an embodiment, the user-specific summarization program 122 selects a subset of the plurality of audio frames and/or video frames based on whether the audio frames and/or video frames contribute to preparing a summary for a user who was away from the conference. In an embodiment, in response to the user-specific summarization program 122 identifying the context of the plurality of audio frames and/or video frames, the user-specific summarization program 122 selects a subset of the plurality of audio frames and/or video frames.

In step 425, the user-specific summarization program 122 ranks the subset of audio frames and/or video frames selected in step 420. In an embodiment, the user-specific summarization program 122 ranks the subset of audio frames. In another embodiment, the user-specific summarization program 122 ranks the subset of video frames. In another embodiment, the user-specific summarization program 122 ranks the subset of audio and video frames.

In embodiments, the user-specific summarization program 122 ranks a subset of the plurality of audio frames and/or video frames using an algorithmic method. In one or more embodiments, the user-specific summarization program 122 ranks a subset of the plurality of audio frames and/or video frames based on whether the audio frame and/or video frame contains the name of a user who was absent from the conference. In one or more embodiments, the user-specific summarization program 122 ranks a subset of the plurality of audio frames and/or video frames based on ranking the participants received in step 405. In one or more embodiments, the user-specific summarization program 122 ranks a subset of the plurality of audio frames and/or video frames based on the relationship between the participant and the user who was absent from the conference. A participant's relevance to the user who was away from the conference is determined by a set of factors including, but not limited to, the participant's role in the conference (i.e., conference organizer, conference required attendee, conference optional attendee), the participant's company profile, the participant's company hierarchy, the participant's position within the company hierarchy, the participant's seniority within the participant's company, the participant's primary job, and the participant's primary job function. In one or more embodiments, the user-specific summarization program 122 ranks a subset of the plurality of audio frames and/or video frames based on the user's technology interests. In one or more embodiments, the user-specific summarization program 122 ranks a subset of the plurality of audio frames and/or video frames based on the user's expertise. In one or more embodiments, the user-specific summarization program 122 ranks a subset of the plurality of audio frames and/or video frames based on past analysis. Although examples of the user-specific summarization program 122 ranking a subset of multiple audio and/or video frames are described herein using separate methods, it should be noted that the user-specific summarization program 122 may rank a subset of multiple audio and/or video frames via one or more combinations of the above embodiments.

In an embodiment, in response to the user-specific summarization program 122 selecting a subset of the plurality of audio frames and/or video frames, the user-specific summarization program 122 ranks the selected subset of the plurality of audio frames and/or video frames in step 420.

At step 430, the user-specific summarization program 122 prepares an integrated summary. In an embodiment, the user-specific summarization program 122 prepares an integrated summary of multiple audio frames ranked above an algorithmically determined threshold. In another embodiment, the user-specific summarization program 122 prepares an integrated summary of multiple video frames ranked above an algorithmically determined threshold. In another embodiment, the user-specific summarization program 122 prepares an integrated summary of multiple audio and video frames ranked above an algorithmically determined threshold.

In an embodiment, the user-specific summarization program 122 prepares the integrated summary by merging multiple audio and/or video frames ranked above an algorithmically determined threshold. In an embodiment, the user-specific summarization program 122 prepares the integrated summary by sequentially splicing together multiple audio and/or video frames ranked above an algorithmically determined threshold and merging multiple audio and/or video frames ranked above an algorithmically determined threshold. In an embodiment, the user-specific summarization program 122 prepares the integrated summary by discarding multiple audio and/or video frames that are not ranked above the algorithmically determined threshold. The threshold is dynamic and changes for each user of the user-specific summarization program 122. In an embodiment, the user-specific summarization program 122 sets the algorithmically determined threshold in accordance with machine learning on a training data set.

In an embodiment, in response to the user-specific summarization program 122 ranking a subset of the plurality of audio frames and/or video frames, the user-specific summarization program 122 prepares an integrated summary.

In a first example, User A's online virtual class on a virtual collaboration server tool is hosted by Teacher X. Invited speaker Y also joins User A's online virtual class. While User A is disconnected from the online virtual class due to User A's network connection issues, Teacher X presents 25 Microsoft® PowerPoint slides. The first slide Teacher X presents is slide 1. However, Teacher X very briefly refers to slide 25 to explain what Invited speaker Y will present later in the class. Teacher X then returns to and explains slides 2-24 before User A reconnects to User A's online virtual class. In this case, the user-specific summarization program 122 ranks slides 1 and 25 higher than slides 2-24 because slides 1 and 25 contain more content information. Slides 1 and 25 are merged.

In a second example, User B's team meeting on the virtual collaboration server tool is hosted by Team Leader X. While User B is disconnected from the team meeting, Team Leader X presents a video that resonates much better with team members than the technical slides because the video is less technical and more illustrative. In this case, the user-specific summarization program 122 ranks the video higher than the technical slides. The video and the other highly ranked slides are merged together, but the technical slides are canceled.

In step 435, the user-specific summarization program 122 converts the integrated summary into one or more sentences. In an embodiment, the user-specific summarization program 122 converts the integrated summary into one or more sentences in text format. In an embodiment, the user-specific summarization program 122 converts the integrated summary into one or more sentences using an extractive text summarization algorithm. In an embodiment, the user-specific summarization program 122 identifies important sentences and other salient information in the integrated summary. In an embodiment, the user-specific summarization program 122 extracts important sentences and other salient information from the integrated summary. Speech-to-text summarization using the extractive text summarization algorithm occurs for the entire duration of the conference, but keeps a rolling 30-minute window open within the duration. In an embodiment, in response to the user-specific summarization program 122 preparing the integrated summary, the user-specific summarization program 122 converts the integrated summary into one or more sentences.

In step 440, the user-specific summarization program 122 applies speaker diarization. Speaker diarization is the process of dividing the input audio stream into homogeneous segments according to speaker identities. Speaker diarization improves the readability of automatic speech transcription by structuring the audio stream into speaker order and, when used in conjunction with a speaker recognition system, by providing true speaker identities. Speaker diarization is a combination of speaker segmentation and speaker clustering. Speaker segmentation aims to detect speaker change points within an audio stream. Speaker clustering aims to group audio segments together based on speaker characteristics. In an embodiment, the user-specific summarization program 122 detects speaker change points within an audio frame (i.e., speaker segmentation). In an embodiment, the user-specific summarization program 122 groups audio segments together based on speaker characteristics (i.e., speaker clustering). In an embodiment, in response to the user-specific summarization program 122 converting the integrated summary into one or more sentences, the user-specific summarization program 122 applies speaker diarization.

In step 445, the user-specific summarization program 122 ranks the one or more sentences. In an embodiment, the user-specific summarization program 122 ranks the one or more sentences based on a set of factors, including, but not limited to, the speaker's role in the meeting (i.e., host, participant, attendee) and the relevance of the sentence to the meeting's intent and/or topic. In an embodiment, the user-specific summarization program 122 retains the top N most useful sentences, where N may be a user preference set by the user in step 220 or a machine-driven recommendation. In an embodiment, in response to the user-specific summarization program 122 applying speaker diarization, the user-specific summarization program 122 ranks the one or more sentences.

For example, if an important person (e.g., the conference host) spoke 100 sentences during the duration of the user's absence from the conference, the user-specific summarization program 122 would rank all 100 sentences in order of importance. However, because N was set to 25, the user-specific summarization program 122 would only retain the top 25 sentences spoken.

In another example, if an important person (e.g., the conference host) speaks 100 sentences during the duration of the user's absence from the conference, the user-specific summarization program 122 ranks all 100 sentences in order of importance. Because all 100 sentences are important, a machine-driven recommendation is made to the user-specific summarization program 122 to retain all 100 sentences, and so the user-specific summarization program 122 retains all 100 sentences.

In yet another example, if an important person (e.g., the meeting host) speaks only one sentence, the user-specific summarization program 122 retains that one sentence.

In step 450, the user-specific summarization program 122 identifies one or more keywords. In an embodiment, the user-specific summarization program 122 identifies one or more keywords in one or more sentences. In an embodiment, the user-specific summarization program 122 identifies one or more keywords related to the intent and/or topic of the meeting. In an embodiment, the user-specific summarization program 122 identifies one or more keywords related to two or more participants of the meeting.

In an embodiment, the user-specific summarization program 122 assigns a weighting to one or more keywords. In an embodiment, the user-specific summarization program 122 assigns a weighting to one or more keywords based on the relevance of the keyword to the intent and/or topic of the meeting. In an embodiment, the user-specific summarization program 122 assigns a weighting to one or more keywords based on the relevance of the keyword to two or more participants of the meeting.

For example, three sentences S10, S11, and S12 are candidates for multiple participants in a meeting. A ranking system is constructed to rank such sentences based on the frequency of keywords that match specific participants. Keywords are identified from S10, S11, and S12. The identified keywords are "data," "algorithm," and "K-means clustering." Participant X10 is a data scientist. Participant X11 is a solution architect. X10 is more interested in sentences containing the keywords "data," "algorithm," and "K-means clustering," while X11 is only interested in sentences containing the keyword "data."

In an embodiment, the user-specific summarization program 122 tags one or more sentences. In an embodiment, the user-specific summarization program 122 tags the one or more sentences using a tagging system. In an embodiment, the user-specific summarization program 122 tags the one or more sentences with words or phrases that express the intent of the one or more sentences. In an embodiment, in response to the user-specific summarization program 122 ranking the one or more sentences, the user-specific summarization program 122 identifies one or more keywords.

For example, three sentences S1, S2, and S3 are spoken with the purpose of motivating or attracting the attention of a group of people. A tagging system is used to tag these sentences as "motivating" or "attention-grabbing". Three sentences S4, S5, and S6 are spoken with the purpose of lightening the mood of a group of people. A tagging system is used to tag these sentences as "lightening the mood". Three sentences S7, S8, and S9 are spoken without any purpose. A tagging system is used to tag these sentences as "nonsense".

In step 455, the user-specific summarization program 122 prepares a global ranking. In an embodiment, the user-specific summarization program 122 prepares a global ranking that incorporates the two or more participants identified and assigned weights in step 405, the one or more keywords identified and assigned weights in step 450, and the subset of one or more sentences ranked in step 455. In an embodiment, the user-specific summarization program 122 allows a user away from the conference to override the ranking priority to assist the user-specific summarization program 122 in preparing a summary that is more tailored to the user. In an embodiment, in response to the user-specific summarization program 122 identifying one or more keywords, the user-specific summarization program 122 prepares a global ranking.

In step 460, the user-specific summarization program 122 prepares summaries. In embodiments, the user-specific summarization program 122 prepares summaries using a machine learning algorithm. In embodiments, the user-specific summarization program 122 prepares summaries for each participant assigned a weight in step 405. In one or more embodiments, the user-specific summarization program 122 prepares summaries for the top N ranked participants, where N may be a user preference or machine-driven recommendation set by the user in step 220.

For example, 15 people (i.e., X1, X2, ..., X15) participate in a conference. The user-specific summarization program 122 assigns a weighting to each participant based on the participant's role in the conference, the participant's role in the participant's enterprise, and the participant's relevance to the user away from the conference. The user-specific summarization program 122 prepares a summary for the top five ranked participants (i.e., X1, X2, X3, X4, and X5).

In an embodiment, the user-specific summarizing program 122 tailors each prepared summary to the profile of the user who left the conference. In an embodiment, the user-specific summarizing program 122 tailors each summary using the global ranking prepared in step 455. In an embodiment, the user-specific summarizing program 122 tailors each summary to proportionally match the relationship between the participant who spoke during the conference and the user who left the conference. The participant's relevance to the user who left the conference is determined by a set of factors, including, but not limited to, the participant's role in the conference (i.e., the conference organizer, the conference requested attendee, the conference optional attendee), the participant's company profile, the participant's company hierarchy, the participant's position within the company hierarchy, the participant's seniority within the participant's company, the participant's primary job, the participant's primary job function, the user's technology interests, and the user's expertise, whether the participant was mentioned by name by the conference host because the participant was expected to speak or answer questions, and whether the participant spoke during the conference.

In an embodiment, the user-specific summarization program 122 adjusts each summary using the speech recognition confidence score, the intent confidence score, or both. In an embodiment, the user-specific summarization program 122 prepares a summary in response to the user-specific summarization program 122 preparing a global ranking.

Figure 5 is a block diagram of components of a computing device 500 within the distributed data processing environment 100 of Figure 1 in accordance with an embodiment of the present invention. Figure 5 provides merely an example of one implementation and is not intended to imply any limitations with regard to the environments in which various embodiments may be implemented. Many modifications to the illustrated environment may be made.

Computing device 500 includes a communications fabric 502 that provides communication between cache 516, memory 506, persistent storage 508, communications unit 510, and input/output (I/O) interface 512. Communications fabric 502 may be implemented using any architecture designed to pass data and/or control information between processors (such as microprocessors, communications processors, and network processors), system memory, peripherals, and any other hardware components in the system. For example, communications fabric 502 may be implemented using one or more buses or crossbar switches.

Memory 506 and persistent storage 508 are computer-readable storage media. In this embodiment, memory 506 includes random access memory (RAM). In general, memory 506 may include any suitable volatile or non-volatile computer-readable storage medium. Cache 516 is high-speed memory that improves the performance of computer processor 504 by holding recently accessed data and data near the accessed data from memory 506.

Programs may be stored in persistent storage 508 and memory 506 for execution and/or access by one or more of the computer processors 504 via cache 516. In an embodiment, persistent storage 508 includes a magnetic hard disk drive. As an alternative or in addition to a magnetic hard disk drive, persistent storage 508 may include a solid-state hard drive, a solid-state storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

The media used by persistent storage 508 may be removable. For example, a removable hard drive may be used for persistent storage 508. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer to another computer-readable storage medium that is also part of persistent storage 508.

In these examples, communications unit 510 provides for communication with other data processing systems or devices. In these examples, communications unit 510 includes one or more network interface cards. Communications unit 510 may provide communications using either or both physical and wireless communications links. Programs may be downloaded to persistent storage 508 via communications unit 510.

I/O interface 512 allows for the input and output of data with other devices that may be connected to server 120 and/or user computing device 130. For example, I/O interface 512 may provide connection to external devices 518, such as a keyboard, keypad, touch screen, or other suitable input device, or a combination thereof. External devices 518 may also include portable computer-readable storage media, such as thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present invention may be stored on such portable computer-readable storage media and loaded into persistent storage 508 via I/O interface 512. I/O interface 512 also connects to display 520.

Display 520 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

The programs described herein are identified based on the application for which they are implemented in particular embodiments of the invention. However, it should be understood that the names of specific programs herein are used merely for convenience, and thus the invention should not be limited to use with only the particular application identified and/or implied by such names.

The present invention may be a system, a method, or a computer program product, or a combination thereof. The computer program product may include a computer-readable storage medium containing computer-readable program instructions for causing a processor to perform aspects of the present invention.

A computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or ridge-in-groove structures on which instructions are recorded, and any suitable combination thereof. As used herein, computer-readable storage media should not be construed as being ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted over wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing/processing device or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof). This network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface within each computing/processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium within each computing/processing device.

The computer-readable program instructions for carrying out the operations of the present invention may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk®, C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network or a wide area network, or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, to carry out aspects of the present invention, electronic circuits including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to customize the electronic circuitry by utilizing state information in the computer-readable program instructions.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to create a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for performing the functions/operations specified in one or more blocks of the flowcharts and/or block diagrams. These computer-readable program instructions may be stored on a computer-readable storage medium and capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions for performing aspects of the functions/operations specified in one or more blocks of the flowcharts and/or block diagrams.

Computer-readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams, thereby causing a series of operable steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or in the reverse order, depending on the functionality involved. It is also noted that each block in the block diagrams and/or flowchart diagrams, and combinations of blocks included in the block diagrams and/or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations or executes a combination of special-purpose hardware and computer instructions.

The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive and is not limited to the disclosed embodiments. Many modifications and variations will become apparent to those skilled in the art without departing from the scope of the present invention. The terms used herein have been selected to best explain the principles, practical applications, or technical improvements of the embodiments beyond those found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims (20)

1. A computer-implemented method comprising:
detecting, by one or more processors, a user absent from a virtual conference including at least two participants for a preset period of time or for a preset percentage of the total allotted time of the pre-scheduled virtual conference;
retrieving, by one or more processors, from a database a first set of data regarding the user, a second set of data regarding the at least two participants of the virtual conference, and a third set of data regarding a relationship between the user and the at least two participants of the virtual conference;
preparing, by one or more processors, a summary tailored to the user's profile and covering the portion of the virtual meeting during which the user was disconnected;
detecting, by one or more processors, that the user reconnects to the virtual conference;
determining, by one or more processors, whether the user will review the summary before rejoining the virtual conference based on preset user preferences, decisions made by the user, or machine-driven recommendations;
prompting, by one or more processors, the user to review the summary using a set of default user preferences in response to the user determining that the user will review the summary before rejoining the virtual conference;
and outputting, by one or more processors, the summary to the user. after outputting the summary to the user, comparing, by one or more processors, the summary with a playback of the complete recording of the virtual conference;
requesting, by one or more processors, feedback from the user;
receiving, by one or more processors, the feedback from the user;
improving, by one or more processors, the accuracy of the prepared future summaries using reinforcement learning;
2. The computer-implemented method of claim 1, further comprising: storing, by one or more processors, the feedback from the user in the database. preparing the summary tailored to the profile of the user and covering the portion of the virtual meeting while the user was disconnected;
identifying, by one or more processors, the at least two participants of the virtual conference;
assigning, by one or more processors, weights to the at least two participants of the virtual conference based on a first set of factors;
and ranking, by one or more processors, the at least two participants of the virtual conference based on the weightings assigned to the at least two participants. extracting, by one or more processors, a plurality of audio frames, a plurality of video frames, or a plurality of audio and video frames from the portion of the virtual conference during which the users were disconnected, after ranking the at least two participants of the virtual conference based on the weights assigned to the at least two participants;
4. The computer-implemented method of claim 3, further comprising: identifying, by one or more processors, a context of the plurality of audio frames, the plurality of video frames, or the plurality of audio and video frames to understand one or more topics of the virtual conference. After identifying the context of the plurality of audio frames, the plurality of video frames, or the plurality of audio and video frames to understand the one or more topics of the virtual conference, selecting, by one or more processors, a subset of the plurality of audio frames, a subset of the plurality of video frames, or a subset of the plurality of audio and video frames based on whether the plurality of audio frames, the plurality of video frames, or the plurality of audio and video frames contribute to preparing a summary for a user who was away from the conference;
ranking, by one or more processors, the subset of the plurality of audio frames, the subset of the plurality of video frames, or the subset of the plurality of audio and video frames using an algorithmic method or a second set of factors;
merging, by one or more processors, the subset of the plurality of audio frames, the subset of the plurality of video frames, or the subset of the plurality of audio and video frames ranked above an algorithmically determined threshold;
5. The computer-implemented method of claim 4, further comprising preparing, by one or more processors, an integrated summary of the subset of the plurality of audio frames ranked above the algorithmically determined threshold, an integrated summary of the subset of the plurality of video frames ranked above the algorithmically determined threshold, and an integrated summary of the subset of the plurality of audio and video frames ranked above the algorithmically determined threshold. after preparing an integrated summary of the subset of the plurality of audio frames ranked above an algorithmically determined threshold, an integrated summary of the subset of the plurality of video frames ranked above an algorithmically determined threshold, and an integrated summary of the subset of the plurality of audio and video frames ranked above an algorithmically determined threshold, converting, by one or more processors, the integrated summaries into one or more sentences using an extractive text summarization algorithm;
applying, by one or more processors, speaker diarization;
ranking, by one or more processors, the one or more sentences based on a third set of factors; and
and retaining, by one or more processors, a subset of the one or more sentences ranked above a second threshold based on second preset user preferences or second machine-driven recommendations. identifying, by one or more processors, one or more keywords in the subset of one or more sentences after retaining the subset of one or more sentences ranked above the second threshold based on the second preset user preferences or the second machine-driven recommendations;
assigning, by one or more processors, a weight to the one or more keywords in the subset of the one or more sentences based on relevance of the one or more keywords to the one or more topics of the virtual conference and based on relevance of the one or more keywords to the at least two participants of the virtual conference;
and tagging, by one or more processors, the subset of the one or more sentences with the one or more keywords. preparing, by one or more processors, a global ranking incorporating the at least two participants, the one or more keywords, and the subset of the one or more sentences after tagging the subset of the one or more sentences with the one or more keywords;
and preparing, by one or more processors, the summary tailored to the user's profile based on the global ranking. The computer-implemented method of claim 1, wherein the first set of data about the user, the second set of data about the at least two participants of the virtual conference, and the third set of data about the relationships between the user and the at least two participants of the virtual conference collected from the database include a profile created by the user, a company profile of the user, a calendar of the user, one or more profiles created by the at least two participants of the virtual conference, the company profiles of the at least two participants of the virtual conference, the calendars of the at least two participants of the virtual conference, one or more presentation tools used by the at least two participants of the virtual conference, and data from one or more previous virtual conferences hosted, participated in, or attended by the user. detecting that the user reconnects to the virtual conference;
capturing, by one or more processors, the time during the virtual conference when the user reconnects;
and calculating, by one or more processors, a total duration of time that the user was disconnected from the virtual conference. The computer-implemented method of claim 1, wherein the set of default user preferences for reviewing the summary includes language preferences, viewing mode preferences, audio preferences, and subtitle preferences. comparing the summary to the playback of the complete recording of the virtual meeting;
extracting, by one or more processors, a first set of one or more key entities from the summary and a second set of one or more key entities from the full recording of the virtual meeting;
3. The computer-implemented method of claim 2, further comprising: matching, by one or more processors, the first set of one or more key entities from the summary with a second set of one or more key entities from the full recording of the virtual meeting. requesting feedback from the user;
3. The computer-implemented method of claim 2, further comprising: providing, by one or more processors, the user with three or more choices representing a satisfaction level of the user, the three or more choices comprising: dissatisfied, neutral, and satisfied. The computer-implemented method of claim 3, wherein the first set of factors includes the roles of the at least two participants in the virtual conference, the roles of the at least two participants in an enterprise, and the relationship of the at least two participants to the user remote from the virtual conference. The computer-implemented method of claim 5, wherein the algorithmically determined threshold is dynamic and tailored to the training data set. The computer-implemented method of claim 5, wherein the second set of factors includes whether the frame includes the name of the user who has left the virtual conference, the ranking of the at least two participants, and the relationship of the at least two participants to the user who has left the virtual conference based on the user's technology interests, the user's expertise, or past analysis. Applying speaker diarization
detecting, by one or more processors, changes between the at least two participants in the plurality of audio frames;
The computer-implemented method of claim 6 , further comprising: grouping, by one or more processors, segments of audio together based on characteristics of the at least two participants. The computer-implemented method of claim 6, wherein the third set of factors includes roles of the at least two participants in the virtual meeting and relevance of the one or more statements to the one or more topics of the virtual meeting. A computer program causing a computer to execute the method of any one of claims 1 to 18. one or more processors;
one or more computer-readable storage media;
and program instructions collectively stored on said one or more computer-readable storage media for execution by at least one of said one or more processors , said stored program instructions including program instructions for performing the method of any one of claims 1 to 18.