US20170046343A1

US20170046343A1 - System and method for removing contextually identical multimedia content elements

Info

Publication number: US20170046343A1
Application number: US15/296,551
Authority: US
Inventors: Igal RAICHELGAUZ; Karina ODINAEV; Yehoshua Y. Zeevi
Original assignee: Cortica Ltd
Current assignee: Cortica Ltd
Priority date: 2005-10-26
Filing date: 2016-10-18
Publication date: 2017-02-16
Also published as: US11403336B2

Abstract

A system and method for removing contextually identical multimedia content elements. The method includes analyzing a plurality of multimedia content elements to identify at least two multimedia content elements of the plurality of multimedia content elements that are contextually identical; selecting, from among the at least two contextually identical multimedia content elements, at least one optimal multimedia content element; and removing, from a storage, all multimedia content elements of the group of contextually identical multimedia content elements other than the at least one optimal multimedia content element.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Patent Application No. 62/310,742 filed on Mar. 20, 2016. This application is a continuation-in-part of U.S. patent application Ser. No. 14/643,694 filed on Mar. 10, 2015, now pending, which is a continuation of U.S. patent application Ser. No. 13/766,463 filed on Feb. 13, 2013, now U.S. Pat. No. 9,031,999. The Ser. No. 13/766,463 application is a continuation-in-part of U.S. patent application Ser. No. 13/602,858 filed on Sep. 4, 2012, now U.S. Pat. No. 8,868,619. The Ser. No. 13/602,858 application is a continuation of U.S. patent application Ser. No. 12/603,123 filed on Oct. 21, 2009, now U.S. Pat. No. 8,266,185. The Ser. No. 12/603,123 application is a continuation-in-part of:
(1) U.S. patent application Ser. No. 12/084,150 having a filing date of Apr. 7, 2009, now U.S. Pat. No. 8,655,801, which is the National Stage of International Application No. PCT/IL2006/001235 filed on Oct. 26, 2006, which claims foreign priority from Israeli Application No. 171577 filed on Oct. 26, 2005, and Israeli Application No. 173409 filed on Jan. 29, 2006;
(2) U.S. patent application Ser. No. 12/195,863 filed on Aug. 21, 2008, now U.S. Pat. No. 8,326,775, which claims priority under 35 USC 119 from Israeli Application No. 185414, filed on Aug. 21, 2007, and which is also a continuation-in-part of the above-referenced U.S. patent application Ser. No. 12/084,150;
(3) U.S. patent application Ser. No. 12/348,888 filed on Jan. 5, 2009, now pending, which is a continuation-in-part of the above-referenced U.S. patent application Ser. No. 12/084,150 and the above-referenced U.S. patent application Ser. No. 12/195,863; and
(4) U.S. patent application Ser. No. 12/538,495 filed on Aug. 10, 2009, now U.S. Pat. No. 8,312,031, which is a continuation-in-part of the above-referenced U.S. patent application Ser. No. 12/084,150, the above-referenced U.S. patent application Ser. No. 12/195,863, and the above-referenced U.S. patent application Ser. No. 12/348,888.
All of the applications referenced above are hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates generally to the analysis of multimedia content, and more specifically to identifying a plurality of multimedia content elements with respect to context.

BACKGROUND

With the abundance of data made available through various means in general and through the Internet and world-wide web (WWW) in particular, a need to understand likes and dislikes of users has become essential for on-line businesses.
Existing solutions provide various tools to identify user preferences. In particular, some of these existing solutions determine user preferences based on user inputs. These existing solutions actively require an input from the user that indicates the user's interests. However, profiles generated for users based on their inputs may be inaccurate, as the users tend to provide only their current interests, or only partial information due to their privacy concerns.
Other existing solutions passively track user activity through web sites such as social networks. The disadvantage with such solutions is that typically limited information regarding the users is revealed because users provide minimal information due to, e.g., privacy concerns. For example, users creating an account on Facebook® typically provide only the mandatory information required for the creation of the account.
Further, user inputs that may be utilized to determine user preferences may be duplicative. For example, a user may provide multiple images of his or her pet to illustrate that he or she has a user preference related to dogs. Such duplicative user inputs require additional memory usage, and may obfuscate the user's true interests. For example, if the user provides 10 images of his or her pet taken around the same time, the system receiving the images typically stores all 10 images, and any user preferences determined therefrom may appear to disproportionately revolve around pets.
It would therefore be advantageous to provide a solution that overcomes the deficiencies of the prior art.

SUMMARY

A summary of several example embodiments of the disclosure follows. This summary is provided for the convenience of the reader to provide a basic understanding of such embodiments and does not wholly define the breadth of the disclosure. This summary is not an extensive overview of all contemplated embodiments, and is intended to neither identify key or critical elements of all embodiments nor to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more embodiments in a simplified form as a prelude to the more detailed description that is presented later. For convenience, the term “some embodiments” may be used herein to refer to a single embodiment or multiple embodiments of the disclosure.
Certain embodiments disclosed herein include a method for removing contextually identical multimedia content elements. The method comprises analyzing a plurality of multimedia content elements to identify at least two multimedia content elements of the plurality of multimedia content elements that are contextually identical; selecting, from among the at least two contextually identical multimedia content elements, at least one optimal multimedia content element; and removing, from a storage, all multimedia content elements of the group of contextually identical multimedia content elements other than the at least one optimal multimedia content element.
Certain embodiments disclosed herein also include a non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute a method, the method comprising: analyzing a plurality of multimedia content elements to identify at least two multimedia content elements of the plurality of multimedia content elements that are contextually identical; selecting, from among the at least two contextually identical multimedia content elements, at least one optimal multimedia content element; and removing, from a storage, all multimedia content elements of the group of contextually identical multimedia content elements other than the at least one optimal multimedia content element.
Certain embodiments disclosed herein also include system for removing contextually identical multimedia content elements. The system comprises: a processing circuitry; and a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to: analyze a plurality of multimedia content elements to identify at least two multimedia content elements of the plurality of multimedia content elements that are contextually identical; select, from among the at least two contextually identical multimedia content elements, at least one optimal multimedia content element; and remove, from a storage, all multimedia content elements of the group of contextually identical multimedia content elements other than the at least one optimal multimedia content element.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter that is regarded disclosed herein is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of the disclosed embodiments will be apparent from the following detailed description taken in conjunction with the accompanying drawings.

FIG. 1 is a network diagram utilized to describe the various embodiments disclosed herein.

FIG. 2 is a schematic diagram of a system for removing contextually identical multimedia content elements according to an embodiment.

FIG. 3 is flowchart illustrating a method for identifying contextually identical multimedia content elements according to an embodiment.

FIG. 4 is a flowchart illustrating a method for generating contextual insights according to an embodiment.

FIG. 5 is a block diagram depicting the basic flow of information in the signature generator system.

FIG. 6 is a diagram showing the flow of patches generation, response vector generation, and signature generation in a large-scale speech-to-text system.

DETAILED DESCRIPTION

It is important to note that the embodiments disclosed herein are only examples of the many advantageous uses of the innovative teachings herein. In general, statements made in the specification of the present application do not necessarily limit any of the various claimed embodiments. Moreover, some statements may apply to some inventive features but not to others. In general, unless otherwise indicated, singular elements may be in plural and vice versa with no loss of generality. In the drawings, like numerals refer to like parts through several views.
Certain embodiments disclosed herein include a system and method for determining whether multimedia content elements are contextually identical. A plurality of multimedia content elements to identify contextually identical multimedia content elements. In an embodiment, the analysis includes generating at least one signature for each multimedia content element. In a further embodiment, the analysis includes matching among the generated signatures to identify signatures representing multimedia content elements that are contextually identical. In another embodiment, the analysis may include determining contextual identifiers for the plurality of multimedia content elements.
Contextually identical multimedia content elements are multimedia content elements associated with the same or nearly the same content. Contextually identical multimedia content elements may be determined to be contextually identical based on, e.g., features of the multimedia content elements (e.g., people and things captured in an image or video, sounds in audio or video, etc.), contextual insights related to the multimedia content elements (e.g., time of capture or receipt, location of capture, device which captured the multimedia content elements, etc.), and the like. For example, two images taken at a concert of a singer that were captured by two users standing next to each other may be contextually identical. As another example, two audio recordings of a song performed by the singer captured at different locations in the concert venue may be contextually identical.
Removing contextually identical multimedia content elements may be useful for, e.g., eliminating duplicative multimedia content elements or multimedia content elements that otherwise include essentially the same content. This elimination may reduce the amount of storage space needed and allows for removal of unnecessary duplicate multimedia content elements. For example, if a user accidentally presses the “capture” button on a camera multiple times when trying to take a picture of a group of friends, multiple images showing essentially the same scene will be captured. As another example, multiple people in a social media group may store multiple instances of the same video. In either example, a essentially duplicate identical multimedia content elements.
In an embodiment, upon identification of contextually identical multimedia content elements, a notification may be generated and sent. In another embodiment, at least one optimal multimedia content element may be determined from among the contextually identical multimedia content elements. The notification may also include a recommendation of the determined at least one optimal multimedia content element. The optimal multimedia content element may be determined based on, but not limited to, features of the multimedia content elements (e.g., resolution, focus, clarity, frame, texture, etc.); matching with other multimedia content elements (e.g., multimedia content elements ranked highly in a social network or liked by a particular user); a combination thereof; and the like. In some embodiments, multimedia content elements that are contextually identical to the optimal multimedia content element may be removed from, e.g., a storage.
As a non-limiting example, a user of a user device captures a series of 10 images determined as self-portrait photographs, which are typically referred to as “selfies”, within a time span of a few minutes. The selfie images are analyzed. In this example, the images are analyzed by at least generating and matching signatures. Based on the analysis, it is determined that the 10 images are contextually identical. Upon determining that the 10 images are contextually identical, an optimal image from among the 10 images is determined and a recommendation of the optimal image is provided. Upon receiving a gesture from a user responsive to the recommendation, images of the contextually identical selfie images other than the optimal image are removed from the storage.
FIG. 1 shows an example network diagram 100 utilized to describe the various embodiments disclosed herein. As illustrated in FIG. 1, a network 110 is communicatively connected to a plurality of user devices (UDs) 120-1 through 120-n (hereinafter referred to individually as a user device 120 and collectively as user devices 120, merely for simplicity purposes), a server 130, a plurality of data sources (DSs) 150-1 through 150-m (hereinafter referred to individually as a data source 150 and collectively as data sources 150, merely for simplicity purposes), and a database 160. In an embodiment, the network 110 may also be communicatively connected to a signature generator system 140. The network 110 may be the Internet, the world-wide-web (WWW), a local area network (LAN), a wide area network (WAN), a metro area network (MAN), and other networks capable of enabling communication between the elements of the system 100.
The user device 120 may be, but is not limited to, a personal computer (PC), a personal digital assistant (PDA), a mobile phone, a tablet computer, a smart phone, a wearable computing device, and the like. Each user device 120 may have installed therein an agent 125-1 through 125-n (hereinafter referred to individually as an agent 125 and collectively as agents 125, merely for simplicity purposes), respectively. The agent 125 may be a dedicated application, script, or any program code stored in a memory (not shown) of the user device 120 and is executable, for example, by the operating system (not shown) of the user device 120. The agent 120 may be configured to perform some or all of the processes disclosed herein.
The user device 120 is configured to capture multimedia content elements, to receive multimedia content elements, to display multimedia content elements, or a combination thereof. The multimedia content elements displayed on the user device 120 may be, e.g., downloaded from one of the data sources 150, or may be embedded in a web-page displayed on the user device 120. Each of the data sources 150 may be, but is not limited to, a server (e.g., a web server), an application server, a data repository, a database, a website, an e-commerce website, a content website, and the like. The multimedia content elements can be locally saved in the user device 120 or can be captured by the user device 120.
For example, the multimedia content elements may include an image captured by a camera (not shown) installed in the user device 120, a video clip saved in the device, an image received by the user device 120, and so on. A multimedia content element may be, but is not limited to, an image, a graphic, a video stream, a video clip, an audio stream, an audio clip, a video frame, a photograph, an image of signals (e.g., spectrograms, phasograms, scalograms, etc.), a combination thereof, a portion thereof, and the like.
The various embodiments disclosed herein may be realized using the server 130, a signature generator system (SGS) 140, or both.
In an embodiment, a tracking agent such as, for example, the agent 125, may be configured to collect and send a plurality of multimedia content elements captured or displayed by the user device 120 to the server 130. In an embodiment, the server 130 may be configured to receive the collected multimedia content elements and to analyze the received multimedia content elements to determine whether and which of the multimedia content elements are contextually identical. The analysis may be based on, but is not limited to, signatures generated for each multimedia content element, concepts determined based on the multimedia content elements, contextual insights for each multimedia content element, a combination thereof, and the like.
In an embodiment, the server 130 is configured to preprocess the multimedia content elements to determine similarities between multimedia content elements of the plurality of multimedia content elements, and only multimedia content elements having similarities above a predetermined threshold are analyzed to determine contextually identical multimedia content elements. In an embodiment, the preprocessing may include analyzing factors including any of the signatures generated for each multimedia content element, the concepts determined based on the multimedia content elements, and the contextual insights for each multimedia content element before analyzing the other factors. For example, it may first be checked if the multimedia content elements were captured within a time period below a predetermined threshold and, if not, the multimedia content elements may be determined not to be contextually identical without generating signatures or determining concepts.
In an embodiment, the server 130 may be configured to send the received multimedia content elements to the signature generator system 140. In an embodiment, the signature generator system 140 is configured to generate at least one signature for each of the multimedia content elements. The process for generating the signatures is explained in more detail herein below with respect to FIGS. 5 and 6. The generated signatures may be robust to noise and distortions as discussed further herein below.
In a further embodiment, the server 130 is further configured to receive the generated signatures from the signature generator system 140. In another embodiment, the server 130 may be configured to generate the at least one signature for each multimedia content element or portion thereof as discussed further herein below.
In an embodiment, whether multimedia content elements are contextually identical may be based on matching between signatures of the multimedia content elements. In a further embodiment, if the matching between the signatures is above a predetermined threshold, the signatures may be determined to be contextually identical.
It should be appreciated that signatures may be used for profiling the user's interests, because signatures allow more accurate recognition of multimedia content elements in comparison to, for example, utilization of metadata. The signatures generated by the signature generator system 140 for the multimedia content elements allow for recognition and classification of multimedia elements such as content-tracking, video filtering, multimedia taxonomy generation, video fingerprinting, speech-to-text, audio classification, element recognition, video/image search, and any other application requiring content-based signatures generation and matching for large content volumes such as web and other large-scale databases. For example, a signature generated by the signature generator system 140 for a picture showing a car enables accurate recognition of the model of the car from any angle at which the picture was taken.
In yet a further embodiment, the server 130 may be configured to match the generated signatures against a database of concepts (not shown) to identify a concept that can be associated with each signature, and hence the corresponding multimedia element.
A concept is a collection of signatures representing at least one multimedia content element and metadata describing the concept. The collection of signatures is a signature reduced cluster generated by inter-matching signatures generated for the at least one multimedia content element represented by the concept. The concept is represented using at least one signature. Generating concepts by inter-matching signatures is described further in U.S. patent application Ser. No. 14/096,901, filed on Dec. 4, 2013, assigned to the common assignee, which is hereby incorporated by reference.
In a further embodiment, matching the generated signatures against the database of concepts further includes matching the generated signatures to signatures representing the concepts. The signatures representing the concepts may be, but are not limited to, signatures included in the concepts or signature clusters representing the concepts.
In an embodiment, whether multimedia content elements are contextually identical may be based at least in part on whether the multimedia content elements are associated with the same or similar concepts. In a further embodiment, determining whether multimedia content elements are associated with the same or similar concepts may be utilized to preprocess and determine multimedia content elements that are not likely contextually identical. That is, in an embodiment, if two or more multimedia content elements are not associated with a similar concept, other factors for determining whether they are contextually identical (e.g., matching between signatures of the multimedia content elements or determination of contextual identifiers) may not be performed. As an example, if a first image is associated with concepts of “books” and “library” while a second image is associated with concepts of “flowers” and “sidewalk”, the first image and the second image may be determined to not be contextually identical without requiring matching between signatures of the first and second images or consideration of time and location of capture of the images.
In another embodiment, the server 130 is further configured to generate at least one contextual insight of the received multimedia content elements. Contextual insights are conclusions related to the context of each multimedia content element, in particular relative to other contexts. In a further embodiment, the contextual insights may be based on metadata associated with each multimedia content element. To this end, in an embodiment, the server 130 is configured to parse the multimedia content elements to determine metadata associated with each multimedia content element.
The metadata may include, but is not limited to, a time pointer associated with a capture or display of a multimedia content element, a location pointer associated with a capture of a multimedia content element, details related to a device (e.g., the user device 120) that captured the multimedia content element, combinations thereof, and the like. In an embodiment, multimedia content elements may be contextually identical if the multimedia content elements were captured or displayed by the same user device 120, at the same (or roughly the same time), at the same (or roughly the same) location, or a combination thereof. Multimedia content elements may be captured or displayed at roughly the same time or location if a difference in the time or location between captures or displays is below a predetermined threshold. For example, if 15 images were captured within a time period of 30 seconds, the 15 images may be determined to be contextually identical. As another example, if two images were captured within 15 feet of each other, the two images may be determined to be contextually identical.
Based on the analysis, the server 130 is configured to determine whether at least two of the received multimedia content elements are contextually identical. As noted above, multimedia content elements may be contextually identical if, for example, signatures of the multimedia content elements match above a predetermined threshold; the multimedia content elements are associated with the same or similar concepts; contextual insights of the multimedia content elements indicate that the multimedia content elements were captured, displayed, or received at the same or similar time; the contextual insights indicate that the multimedia content elements were captured at the same or similar location; the contextual insights indicate that the multimedia content elements were captured by the same device; or a combination thereof.
In an embodiment, when it is determined that at least two multimedia contents are contextually identical, the server 130 is configured to send a notification indicating the at least two contextually identical multimedia content elements. In a further embodiment, the server 130 may be configured to receive a selection of one of the at least two contextually identical multimedia content elements. In yet a further embodiment, the server 130 is configured to remove, from a storage (e.g., one of the data sources 160), multimedia content elements of the at least two multimedia content elements other than the selected multimedia content element. Removing unselected contextually identical multimedia content elements reduces
In a further embodiment, the server 130 may be configured to determine at least one optimal multimedia content element from among the at least two contextually identical multimedia content elements. The at least one optimal multimedia content element is a multimedia content element selected to represent the at least two contextually identical multimedia content elements. The at least one optimal multimedia content element may be determined based on, but not limited to, features of the multimedia content elements (e.g., resolution, focus, clarity, frame, texture, etc.); matching with other multimedia content elements (e.g., multimedia content elements ranked highly in a social network or liked by a particular user); a combination thereof; and the like.
In a further embodiment, the server 130 is configured to determine the optimal multimedia content based on, but not limited to, matching between signatures representing the at least two contextually identical multimedia content elements and signatures representing concepts a particular user is interested in. In yet a further embodiment, the contextually identical multimedia content element having the signature with the highest matching to the user interest concept signatures may be determined as the optimal multimedia content element.
To this end, each concept may be associated with at least one user interest. For example, a concept of flowers may be associated with a user interest in ‘flowers’ or ‘gardening.’ In an embodiment, the user interest may simply be the identified concept. In another embodiment, the user interest may be determined using an association table which associates one or more identified concepts with a user interest. For example, the concepts of ‘flowers’ and ‘spring’ may be associated, in an association table with a user interest of ‘gardening’. Such an association table may be maintained in, e.g., the server 130 or the database 160.
In an embodiment, the notification may further indicate the at least one optimal multimedia content element. In a further embodiment, the notification including the at least one optimal multimedia content element is then provided to the user device 120 and the user device 120 is prompted to confirm selection of the at least one optimal multimedia content element. When the selection is confirmed, the server 130 is configured to remove the multimedia content element(s) of the at least two contextually identical multimedia content elements which were not determined as optimal from, e.g., a storage. In an embodiment, the server 130 is configured to remove the non-optimal multimedia content elements in real-time. In another embodiment, the server 130 may be configured to automatically remove the non-optimal multimedia content elements when at least one optimal multimedia content element is determined.
Each of the server 130 and the signature generator system 140 typically includes a processing circuitry (not shown) that is coupled to a memory (not shown). The memory typically contains instructions that can be executed by the processing circuitry. The server 130 also includes an interface (not shown) to the network 110. In an embodiment, the signature generator system 140 can be integrated in the server 130. In an embodiment, the server 130, the signature generator system 140, or both may include a plurality of computational cores having properties that are at least partly statistically independent from other of the plurality of computational cores. The computational cores are discussed further herein below.
FIG. 2 is an example schematic diagram of a system for removing contextually identical multimedia content elements according to an embodiment. In the example schematic diagram shown in FIG. 2, the system is the server 130. It should be noted that, in another embodiment, the system may be the user device 120. In a further embodiment, the agent 125 installed on the user device 120 may be configured to identify contextually identical multimedia content elements as described herein.
The server 130 includes an interface 210 at least for receiving multimedia content elements captured or displayed by the user device 120 and for sending notifications indicating contextually identical multimedia content elements, optimal multimedia content elements, or both, to the user device 120. The server 130 further includes a processing circuitry 220 such as a processor coupled to a memory (mem) 230. The memory 230 contains instructions that, when executed by the processing circuitry 220, configures the server 130 to identify contextually identical multimedia content elements as further described herein.
In an embodiment, the server 130 also includes a signature generator (SG) 240. The signature generator 240 includes a plurality of computational cores having properties that are at least partly statistically independent from other of the plurality of computational cores. The signature generator 240 is configured to generate signatures for multimedia content elements. In an embodiment, the signatures are robust to noise, distortion, or both. In another embodiment, the server 130 may be configured to send, to an external signature generator (e.g., the signature generator system 140), one or more multimedia content elements and to receive, from the external signature generator, signatures generated to the sent one or more multimedia content elements.
In another embodiment, the server 130 includes a data storage 250. The data storage may store, for example, signatures of multimedia content elements, signatures of concepts, contextually identical multimedia content elements, optimal multimedia content elements, combinations thereof, and the like.
FIG. 3 is an example flowchart 300 illustrating a method for identifying and removing contextually identical multimedia content elements (MMCEs) according to an embodiment. In an embodiment, the method may be performed by the server 130, the user device 120, or both. In an embodiment, the contextually identical multimedia content elements are identified based on a plurality of received multimedia content elements. The received multimedia content elements may be, e.g., multimedia content elements captured by a user device, multimedia content elements stored on a server (e.g., a server of a social network entity), and so on.
At optional S310, the plurality of multimedia content elements may be preprocessed. The preprocessing allows for, e.g., reduced usage of computing resources. To this end, in an embodiment, S310 includes, but is not limited to, determining at least one contextual insight (e.g., time, location, or device of capture or display) for each of the plurality of multimedia content elements, determining a concept associated with each of the plurality multimedia content elements, or both. Determining contextual insights and concepts for multimedia content elements are described further herein above with respect to FIG. 1. In a further embodiment, S310 further includes determining, based on the concepts, contextual insights, or both, whether any of the plurality of multimedia content elements are potentially contextually identical. In yet a further embodiment, S310 may include filtering out any of the multimedia content elements that are not determined to be potentially contextually identical.
At S320, the multimedia content elements are analyzed to identify at least one group of contextually identical multimedia content elements. Each group of contextually identical multimedia content elements includes at least two multimedia content elements that are contextually identical to each other. In an embodiment, the analysis may be based on, but not limited to, at least one contextual insight of each multimedia content element, at least one concept associated with each multimedia content element, at least one signature of each multimedia content element, or a combination thereof. Analyzing multimedia content elements to identify contextually identical multimedia content elements is described further herein below with respect to FIG. 4.
In another embodiment, S320 may include sending, to a signature generator system (e.g., the signature generator system 140) the multimedia content elements and receiving, from the signature generator system, at least one signature for each sent multimedia content element.
At S330, it is determined, based on the analysis, whether any multimedia content elements were identified as being contextually identical to each other. If so, execution continues with S340; otherwise, execution terminates.
At S340, at least one optimal multimedia content element may be determined from among the identified contextually identical multimedia content elements. In an embodiment, the at least one optimal multimedia content element may be determined based on, but not limited to, features of the multimedia content elements (e.g., resolution, focus, clarity, frame, texture, etc.); matching with other multimedia content elements (e.g., multimedia content elements ranked highly in a social network or liked by a particular user); a combination thereof; and the like.
In a further embodiment, one optimal multimedia content element may be selected for each group of contextually identical multimedia content elements that are contextually identical to each other. As an example, if the plurality of multimedia content elements includes 3 images showing a dog that are contextually identical and 5 videos showing a cat that are contextually identical, an optimal image may be selected from among the 3 contextually identical dog images and an optimal video may be selected from among the 5 contextually identical cat videos.
At S350, for each group of contextually identical multimedia content elements, all multimedia contents of the set other than the at least one optimal multimedia content are removed from, e.g., a storage. The removal may be automatic and in real-time. Alternatively, in another embodiment, S350 may include sending, to a user device, a notification indicating the selecting optimal multimedia content elements and prompting a user to confirm selection of the optimal multimedia content elements. In a further embodiment, upon receiving confirmation of the selection of the optimal multimedia content elements, S350 includes automatically removing all non-optimal multimedia content elements. In yet a further embodiment, S350 may further include receiving a selection of at least one alternative optimal multimedia content element. In such an embodiment, all multimedia content elements other than the at least one alternative optimal multimedia content may be removed from the storage.
As a non-limiting example, a plurality of images is received. The plurality of images is stored in a web server of a social network. The plurality of images includes 10 images showing a group of friends and one image showing an ocean. The plurality of images are preprocessed by determining contextual insights for each image. Each image is parsed to identify metadata, and the metadata is analyzed to determine the contextual insights. Based on the contextual insights, it is determined that the image showing the ocean was captured one hour after the images showing the group of friends, and that the images showing the group of friends were captured within 1 minute of each other. Accordingly, the images showing the group of friends are determined to be potentially contextually identical, and the image of the ocean is filtered out.
The remaining images showing the group of friends is analyzed by generating and matching signatures for each of the images. Based on the signature matching, it is determined that all of the images showing the group of friends match above a predetermined threshold. Thus, it is determined that the 10 images of the group of friends are contextually identical. Features of the contextually identical images are analyzed. Based on the feature analysis, it is determined that one of the contextually identical images has a higher resolution than other of the contextually identical images. The higher resolution image is selected as the optimal image, and the other images of the group of friends are removed from the web server.
FIG. 4 is an example flowchart S320 illustrating a method for analyzing a plurality of multimedia content elements to identify contextually identical multimedia content elements according to an embodiment.
At S410, at least one signature for each multimedia element identified is caused to be generated. In an embodiment, S410 may further include sending, to a signature generator system, the plurality of multimedia content elements and receiving, from the signature generator system, signatures generated for the plurality of multimedia content elements. Generation of signatures is described further herein below with respect to FIGS. 5-6.
At S420, the generated signatures are matched. Matching between signatures is described further herein below with respect to FIG. 5.
At S430, it is determined, based on the signature matching, whether any of the plurality of multimedia content elements are contextually identical and, if so, execution continues with S440; otherwise, execution terminates. In an embodiment, S430 includes determining, based on the matching, whether signatures representing any of the plurality of multimedia content elements match above a predefined threshold, where two or more multimedia content elements are contextually identical to each other when signatures representing the two or more multimedia contents match above a predetermined threshold.
At S440, when it is determined that at least two of the multimedia content elements are contextually identical, at least one group of contextually identical multimedia content elements is identified. Each set includes at least two multimedia content elements that are contextually identical to each other.
FIGS. 5 and 6 illustrate the generation of signatures for the multimedia elements by the signature generator system 140 according to one embodiment. An exemplary high-level description of the process for large scale matching is depicted in FIG. 5. In this example, the matching is for a video content.
Video content segments 2 from a Master database (DB) 6 and a Target DB 1 are processed in parallel by a large number of independent computational Cores 3 that constitute an architecture for generating the Signatures (hereinafter the “Architecture”). Further details on the computational Cores generation are provided below. The independent Cores 3 generate a database of Robust Signatures and Signatures 4 for Target content-segments 5 and a database of Robust Signatures and Signatures 7 for Master content-segments 8. An exemplary and non-limiting process of signature generation for an audio component is shown in detail in FIG. 5. Finally, Target Robust Signatures and/or Signatures are effectively matched, by a matching algorithm 9, to Master Robust Signatures and/or Signatures database to find all matches between the two databases.
To demonstrate an example of signature generation process, it is assumed, merely for the sake of simplicity and without limitation on the generality of the disclosed embodiments, that the signatures are based on a single frame, leading to certain simplification of the computational cores generation. The Matching System is extensible for signatures generation capturing the dynamics in-between the frames.
The Signatures' generation process is now described with reference to FIG. 5. The first step in the process of signatures generation from a given speech-segment is to break down the speech-segment to K patches 14 of random length P and random position within the speech segment 12. The breakdown is performed by the patch generator component 21. The value of the number of patches K, random length P and random position parameters is determined based on optimization, considering the tradeoff between accuracy rate and the number of fast matches required in the flow process of the server 130 and SIGNATURE GENERATOR SYSTEM 140. Thereafter, all the K patches are injected in parallel into all computational Cores 3 to generate K response vectors 22, which are fed into a signature generator system 23 to produce a database of Robust Signatures and Signatures 4.
In order to generate Robust Signatures, i.e., Signatures that are robust to additive noise L (where L is an integer equal to or greater than 1) by the Computational Cores 3, a frame ‘i’ is injected into all the Cores 3. Then, Cores 3 generate two binary response vectors: {right arrow over (S)} which is a Signature vector, and {right arrow over (RS)} which is a Robust Signature vector.
For generation of signatures robust to additive noise, such as White-Gaussian-Noise, scratch, etc., but not robust to distortions, such as crop, shift and rotation, etc., a core Ci={ni} (1≦i≦L) may consist of a single leaky integrate-to-threshold unit (LTU) node or more nodes. The node ni equations are:
$V_{i} = \sum_{j} w_{ij} k_{j} ni = θ (Vi - Thx)$
where, θ is a Heaviside step function; wij is a coupling node unit (CNU) between node i and image component j (for example, grayscale value of a certain pixel j); kj is an image component T (for example, grayscale value of a certain pixel j); Thx is a constant Threshold value, where x is ‘S’ for Signature and ‘RS’ for Robust Signature; and Vi is a Coupling Node Value.
The Threshold values ThX are set differently for Signature generation and for Robust Signature generation. For example, for a certain distribution of values (for the set of nodes), the thresholds for Signature (ThS) and Robust Signature (ThRS) are set apart, after optimization, according to at least one or more of the following criteria:

- 1: For: V_i>Th_RS

1−p(V>Th_S)−1−(1−ε)^l<<1
i.e., given that l nodes (cores) constitute a Robust Signature of a certain image I, the probability that not all of these l nodes will belong to the Signature of a same, but noisy image, Ĩ is sufficiently low (according to a system's specified accuracy).

- 2: p(V_i>Th_RS)≈l/L

i.e., approximately l out of the total L nodes can be found to generate a Robust Signature according to the above definition.

- 3: Both Robust Signature and Signature are generated for certain frame i.

It should be understood that the generation of a signature is unidirectional, and typically yields lossless compression, where the characteristics of the compressed data are maintained but the uncompressed data cannot be reconstructed. Therefore, a signature can be used for the purpose of comparison to another signature without the need of comparison to the original data. The detailed description of the Signature generation can be found U.S. Pat. Nos. 8,326,775 and 8,312,031, assigned to common assignee, and are hereby incorporated by reference for all the useful information they contain.
A Computational Core generation is a process of definition, selection, and tuning of the parameters of the cores for a certain realization in a specific system and application. The process is based on several design considerations, such as:

- (a) The Cores should be designed so as to obtain maximal independence, i.e., the projection from a signal space should generate a maximal pair-wise distance between any two cores' projections into a high-dimensional space.
- (b) The Cores should be optimally designed for the type of signals, i.e., the Cores should be maximally sensitive to the spatio-temporal structure of the injected signal, for example, and in particular, sensitive to local correlations in time and space. Thus, in some cases a core represents a dynamic system, such as in state space, phase space, edge of chaos, etc., which is uniquely used herein to exploit their maximal computational power.
- (c) The Cores should be optimally designed with regard to invariance to a set of signal distortions, of interest in relevant applications.

Detailed description of the Computational Core generation and the process for configuring such cores is discussed in more detail in the U.S. Pat. No. 8,655,801 referenced above.
The various embodiments disclosed herein can be implemented as hardware, firmware, software, or any combination thereof. Moreover, the software is preferably implemented as an application program tangibly embodied on a program storage unit or computer readable medium consisting of parts, or of certain devices and/or a combination of devices. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units (“CPUs”), a memory, and input/output interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU, whether or not such a computer or processor is explicitly shown. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit. Furthermore, a non-transitory computer readable medium is any computer readable medium except for a transitory propagating signal.
All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the principles of the disclosed embodiment and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosed embodiments, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

Claims

What is claimed is:

1. A method for removing contextually identical multimedia content elements, comprising:

analyzing a plurality of multimedia content elements to identify at least two multimedia content elements of the plurality of multimedia content elements that are contextually identical;

selecting, from among the at least two contextually identical multimedia content elements, at least one optimal multimedia content element; and

removing, from a storage, all multimedia content elements of the group of contextually identical multimedia content elements other than the at least one optimal multimedia content element.

2. The method of claim 1, wherein the analysis is based on at least one of: contextual insights of the plurality of multimedia content elements, concepts associated with the plurality of multimedia content elements, and signatures of the plurality of multimedia content elements.

3. The method of claim 2, wherein analyzing the plurality of multimedia content elements further comprises:

causing generation of at least one signature for each of the plurality of multimedia content elements; and

matching between the signatures of the plurality of multimedia content elements, wherein the at least two contextually identical contextually identical multimedia content elements are identified based on the signature matching.

4. The method of claim 3, wherein at least two multimedia content elements are contextually identical when signatures of the at least two multimedia content elements match above a predetermined threshold.

5. The method of claim 3, wherein the generation of the at least one signature for each of the plurality of multimedia content elements is caused by a signature generator system including a plurality of at least computational cores, each computational core having properties that are at least partly statistically independent of other of the computational cores, wherein the properties of each core are set independently of each other core.

6. The method of claim 2, wherein analyzing the plurality of multimedia content elements further comprises:

generating, based on metadata associated with each multimedia content element, at least one contextual insight, wherein the analysis is based on the generated at least one contextual insight.

7. The method of claim 2, wherein analyzing the plurality of multimedia content elements further comprises:

causing generation of at least one signature for each of the plurality of multimedia content elements;

determining, based on the generated signatures, at least one concept for each multimedia content element, wherein the analysis is based on the generated concepts, wherein each concept is a collection of signatures and metadata representing the concept.

8. The method of claim 1, wherein the at least one optimal multimedia content element is selected based on at least one of: features of the at least two contextually identical multimedia content elements, and matching between signatures of the at least two contextually identical multimedia content elements with signatures of at least one other multimedia content element.

9. The method of claim 1, further comprising:

preprocessing the plurality of multimedia content elements to identify a plurality of potentially contextually identical multimedia content elements, wherein the at least two contextually identical multimedia content elements are identified from among the plurality of potentially contextually identical multimedia content elements.

10. A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute a method, the method comprising:

11. A system for removing contextually identical multimedia content elements, comprising:

a processing circuitry; and

a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:

analyze a plurality of multimedia content elements to identify at least two multimedia content elements of the plurality of multimedia content elements that are contextually identical;

select, from among the at least two contextually identical multimedia content elements, at least one optimal multimedia content element; and

remove, from a storage, all multimedia content elements of the group of contextually identical multimedia content elements other than the at least one optimal multimedia content element.

12. The system of claim 11, wherein the analysis is based on at least one of: contextual insights of the plurality of multimedia content elements, concepts associated with the plurality of multimedia content elements, and signatures of the plurality of multimedia content elements.

13. The system of claim 12, wherein the system is further configured to:

cause generation of at least one signature for each of the plurality of multimedia content elements; and

match between the signatures of the plurality of multimedia content elements, wherein the at least two contextually identical contextually identical multimedia content elements are identified based on the signature matching.

14. The system of claim 13, wherein at least two multimedia content elements are contextually identical when signatures of the at least two multimedia content elements match above a predetermined threshold.

15. The system of claim 13, wherein the generation of the at least one signature for each of the plurality of multimedia content elements is caused by a signature generator system including a plurality of at least computational cores, each computational core having properties that are at least partly statistically independent of other of the computational cores, wherein the properties of each core are set independently of each other core.

16. The system of claim 12, wherein the system is further configured to:

generate, based on metadata associated with each multimedia content element, at least one contextual insight, wherein the analysis is based on the generated at least one contextual insight.

17. The system of claim 12, wherein the system is further configured to:

cause generation of at least one signature for each of the plurality of multimedia content elements;

determine, based on the generated signatures, at least one concept for each multimedia content element, wherein the analysis is based on the generated concepts, wherein each concept is a collection of signatures and metadata representing the concept.

18. The system of claim 11, wherein the at least one optimal multimedia content element is selected based on at least one of: features of the at least two contextually identical multimedia content elements, and matching between signatures of the at least two contextually identical multimedia content elements with signatures of at least one other multimedia content element.

19. The method of claim 11, wherein the system is further configured to:

preprocess the plurality of multimedia content elements to identify a plurality of potentially contextually identical multimedia content elements, wherein the at least two contextually identical multimedia content elements are identified from among the plurality of potentially contextually identical multimedia content elements.