WO2014076442A1

WO2014076442A1 - A self-service facility for content providers

Info

Publication number: WO2014076442A1
Application number: PCT/GB2012/052832
Authority: WO
Inventors: Kristoffer HAMMER
Original assignee: Clearcast Limited
Priority date: 2012-11-15
Filing date: 2012-11-15
Publication date: 2014-05-22

Abstract

A system that allows users to receive interactive content related to a non-interactive source, such as a newspaper, poster, or label through the use of automatic content recognition (ACR) applications (apps). The user can take a picture of the non-interactive source, submits the picture to the system, and receives interactive content regarding the subject of the picture. The system maintains a searchable library of links to interactive content on a wide range of subjects. The system also provides a self-service facility for content providers to produce the interactive content. The interactive content produced by the content providers is used to populate the searchable library. Recognition software is used to identify the subject of submitted pictures, and matching software searches the library for content that matches the subject. The matched, interactive content is then sent to the user. Interactions with the content, by the user, can be monitored and reported to the content providers in real-time. In an alternate embodiment, the non-interactive source can be a television or radio wherein, the user records and submits an audio or audio/video file to the system. The system recognizes a subject of the file, searches the library for matching content, and sends the matched interactive content to the user. The system does not offer the consumer facing app, but the access to the above library to all ACR apps.

Description

A SELF-SERVICE FACILITY FOR CONTENT PROVIDERS

FIELD OF THE INVENTION

The present invention relates generally to the field of electronic commerce, and more specifically to providing interactive content in response to a picture taken of, or a filed recorded from, a non-interactive source and making this available to applications offering automatic content recognition (ACR) services.

BACKGROUND OF THE INVENTION People are surrounded by non-interactive material everyday, such as household items, clothing, cars, buildings, newspapers, magazines, posters, labels, traditional TV, and radio. Everyday people are intrigued by what they see and hear, and would like to get more information on these non-interactive items. What is needed is a system that allows people to take a picture or record a file of a subject they are interested in, submit the picture or file, and receive interactive material about the subject. Allowing people to receive interactive content regarding a non- interactive item requires recognizing the subject. Thankfully, many different types of recognition software have been developed and are currently available on the market. Shazam is an example of audio recognition software that allows a user to record an audio sample, submit the sample and subsequently be provided with a complete file of the sampled audio. Soundhound is another example of commercially available audio recognition software. Clic2c is an example of visual recognition software that was specially designed for recognizing visual items in magazines. RedLaser is another example of visual recognition software that provides for recognition of pictures of products. Facial recognition software allows a system to automatically identify a person from a digital image or a video frame from a video source.

Commercially available facial recognition software includes FaceSDK from Luxand, BananaSecurity form Xentrik, and REST API from Face.com. Zeebox is an example of recognition software that monitors keywords from the audio streams of television programs. Many of these recognition systems use tagging, watermarks, or fingerprints to recognize a subject. One or more of these recognition software programs can be used with the present system to identify the subject, or subjects, of a submission.

In order to provide users with additional information regarding the subject of their submission, a structured database of content must be provided. A database is an organized collection of data. The data is typically organized to model relevant aspects of reality in a way that supports processes requiring this information.

Structured databases are managed by a database management system (DBMS). Commercially available DBMSs include Oracle, IBM DB2, Microsoft SQL Server, and Microsoft Access.

With the explosion in popularity of cellular telephones, including smart phones, it would be useful to allow users to submit their picture or file from their cell phone. A cellular telephone, or cell phone, is a digital device that can make and receive telephone calls over a radio link. Cell phones do so by connecting to a cellular network provided by a cell phone operator, allowing access to the public telephone network. Thus, cell phones also have access to the Internet. A cellular network or mobile network is a radio network distributed over land areas called cells. Each cell is served by at least one fixed-location transceiver, known as a cell site or base station. In a cellular network, each cell uses a different set of frequencies from neighboring cells, to avoid interference and provide guaranteed bandwidth within each cell. Cell phone tracking refers to the attaining of the current position of a cell phone. Location determination can occur either via multilateration of radio signals between radio towers of the network and the phone, or simply via GPS.

The ideal system for providing interactive content to users would include recognition software that allows pictures, audio and video to be recognized. The system would maintain a large library of interactive content that included information on a wide range of subjects. The library would be stored in a structured database that allows for easy searching and quick retrieval of desired items. The system would accept submission from wired and wireless customers. The ideal system would further allow for easy population of the library by providing a self-service facility to content providers to produce and upload the content to the library.

The library is open for any automatic content recognition (ACR) application which conducts searches in the fields which are covered by this service, i.e., advertising and packaging, editorial content (broadcast and non-broadcast) and user generated content, but not limited to these.

The present invention does not seek to provide ACR technology / searches itself and is therefore different from any of the propositions mentioned above. The present invention seeks to ensure that owners of various kinds of content can better control the results of ACR searches of content they have the ownership to.

SUMMARY OF THE INVENTION

A system that allows a user to take a picture of any visual material, submit the picture to the present Grid system, and receive interactive content regarding the subject of the picture. The Grid system comprises a content server that stores links to interactive content on a wide range of subjects and is adapted to receive the picture from the user. The content server has recognition software that identifies the subject of the picture, matches the subject to content stored therein, and sends the matched content back to the user. The system can also allow for an independent recognition software service to access the library, to search for matches. The content server further includes a user interface that facilitates electronic communications with the user, an administrator interface that facilitates electronic communications with a system administrator, and a content provider interface that facilitates electronic communications with multiple content providers. The content server provides a self- service facility for the content providers, such as advertisers, by providing all tools necessary for the content providers to create links to interactive content and upload the links to interactive content to the content server.

The content server is communicatively coupled to a support server. The support server comprises an automatic content recognition (ACR) module and a metrics module. The automatic content recognition (ACR) module communicates with ACR apps, the user interface and the content provider interface to provide tracking information in real-time regarding user interactions with the interactive content. The metrics module communicates with the ACR module and collects metric data regarding the user interactions and assists in making the metric data available to the content providers in real-time. The metric data is collected on both user submissions and user interactions.

In the preferred embodiment, a user takes a picture of any non-interactive material using a digital device, such as smart phone with a camera. An automatic content recognition (ACR) application (app) in the user's device then submits the picture to the content server. The interactive content received from the content server includes at least one link that when clicked on, or otherwise activated, by the user connects the user to a webpage, or other content, selected by the content provider. The ACR app allows the user to receive and display the interactive content that is linked to the content server. The ACR app also automatically contacts the Grid system with interaction information when the user activates the link, and when the user interacts with the webpage.

The ACR module receives activity reports from the ACR app on the user's device, regarding the user's interactions with the interactive content. The ACR module transfers this information to the metrics module for parsing, storing and distribution. The content provider interface and a downloadable Grid provider Ul allow for access to the metrics module and facilitates viewing of the metrics data in real-time.

The Grid provider Ul and the provider interface also give the content providers access to upload a collection of self-service files, including audio, visual, and audio- video files that content providers can use to produce customized interactive content. For example, a content provider can use an audio-video file from the collection and add content and one or more URLs to the file to produce interactive content. The content provider is able to associate different URLs with different segments of the audio-video file, so that if a link associated with the file is activated at the beginning of the file, a person viewing the file will be taken to a first website, however, if the link is activated at the end of the audio-video file, or in another segment of the file, the person is taken to a second website. The system also allows the user to receive new and additional links as the audio-visual or audio file he is searching progresses. The Grid provider Ul also allows the content providers to include instructions with the produced interactive content, wherein the instructions can provide for different URLs being sent with matched content, based on a user profile, a time of day, a location, or a number of submissions. The Grid provider Ul also facilitates formatting and uploading the links to the produced interactive content to the content server.

Content that is sent to the user can include an audio file, a video file, an audio-video file, an advertisement, a coupon, a gift card, documentary or editorial information regarding the subject of the picture, a text document, a document with hyperlinks, a multi-media document, a link to a webpage, and/or other pictures related to the subject.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described in more detail with reference to the accompanying drawings, given only by way of example, in which:

Figure 1 is an overall view of a preferred embodiment;

Figure 2 shows exemplary hardware and software for the present Grid system;

Figure 3 is a flow chart showing user interactions with the Grid system;

Figure 4 illustrates an exemplary graphical user interface for the Grid system;

Figure 5 is a flow chart showing content provider interactions with the Grid system;

Figure 6 is a flow chart showing more detailed content provider interactions with the Grid system;

Figure 7 illustrates an exemplary content provider interface display for the

Grid system;

Figure 8 is a flow chart showing exemplary communications between a user's device and the Grid system; Figure 9 illustrates another exemplary content provider interface display for the Grid system;

Figure 10 illustrates an alternate embodiment wherein the user submits a file to the Grid system;

Figure 1 1 is a flow chart showing exemplary steps in the alternate

embodiment; and,

Figure 12 shows an exemplary format for an entry in the Grid system.

Figure 13 shows an alternative embodiment wherein data from a file recorded by the user is submitted to an identification server and wherein links to interactive content are received from a content server.

DETAILED DESCRIPTION OF THE INVENTION

The present Grid system is intended to add an interactive feature to non- interactive material. The preferred embodiment provides for taking a picture of any non-interactive material, such as printed material, and being provided with interactive content that is related to the picture. The Grid system allows any material that can be captured by a camera, to be placed on an interactive Grid, meaning that the user is given the ability to interact with links, files, documents and other information available through the Grid. The Grid is stocked with interactive content on a wide range of subjects. The content is loaded onto the Grid by content providers. The Grid system encourages the uploading of content by providing a self-service facility to the content providers. The Grid system also allows content providers to monitor user interactions with their content in real-time.

Referring to Fig. 1 , an overall view of a preferred embodiment is illustrated. The user 100 is a person that is interested in getting more information on an item that they just saw. In this example, the user 100 just saw a picture of the actress Teri Hatcher in a newspaper 102. Teri was photographed wearing a stunning dress at a recent awards show. The user 100 is a big fan of the television show

"Desperate Housewives" and enjoys purchasing dresses similar to those worn by Ms. Hatcher. Using the digital device 101 , the user 100 takes a picture of Ms.

Hatcher from the newspaper 102. In other embodiments, the user takes a picture of an object (such as a watch or car), a label, a poster, or the page of a book or magazine. The digital device 101 is preferably a cellular telephone with a camera. In other embodiments, the digital device is a laptop computer, desktop computer or tablet computer, each with a camera or microphone. Recognition software in the device 101 tags the picture, and the picture is transmitted to the Internet 103 via a wireless communications channel 106. The picture is routed through the Internet 103 and communications channel 107 to the present Grid system 104. The Grid system 104 includes a large collection of interactive content on a wide range of subjects, including TV stars and popular fashion. Using the tags that were added to the picture, the Grid system 104 identifies a subject of the picture; Teri Hatcher at an awards show, in this case. The system 104 then matches interactive content to the picture and sends the interactive content to the user's device 101 . The matched content that is sent to the user 100 may include links to websites that sell the same dress worn by Ms. Hatcher in the picture, websites that sell look-a-like dresses worn by Ms. Hatcher, websites that feature Teri Hatcher, websites dedicated to the show "Desperate Housewives", websites associated with the awards show, and

documents and files related to the subject.

Important parts of the Grid system 104 are the content providers. Multiple content providers can access the Grid system 104 using content provider computers 105. After registering with the system 104 the content providers are provided with all the tools necessary to create, format and link to, interactive content. Content providers are also able to monitor user interactions with their content. Advertisers, for example, can receive all metrics associated with a traditional ad server, so that activations of interactive ads, and purchases of advertised products can be monitored. Content provider computers 105 are connected to the Internet 103 via a digital communications channel 108. While content provider computers 105 are shown physically close to each other, in reality the computers of different content providers will likely be widely distributed.

Referring to Fig. 2, a more detailed view of the Grid system 104 is provided. The primary component of the system is the content server 200. The content server 200 includes a database that houses a library of content 201 with a large collection of links to interactive content classified into multiple subjects. Each entry in the library of content 201 includes at least one link to a webpage, file or document, and the entry is placed in the library 201 according to a classification scheme. The webpage, file or document can be an advertisement, documentary or editorial information, a text document, a document with hyperlinks, a multi-media document, a coupon, a gift card, and/or one or more other pictures related to the subject. The library of content 201 is a structured database that allows for easy search and retrieval of stored content. The applications (Apps) software 202 of the server 200 communicates with the user interface 206 in order to facilitate communications between the system 104 and the automatic content recognition (ACR) application (app) on the user's device 101 . The ACR app is preferably downloaded to the user's device 101 from a third party recognition software provider, such as Shazam, Soundhound, Clic2c and/or RedLaser. The Grid system 104 integrates these third party software recognition apps into the Grid system and allows consumers to use a recognition app that they are familiar with, in order to receive desired interactive content from the system 104. Integration of the third party app includes

supplementing the ACR app with functionality that provides for display of content and monitoring of user interactions with the content. Apps software 202 also communicates with the content provider interface 207 in order to facilitate download of a Grid provider User Interface to each of the content provider computers 105. The Grid provider User Interface gives the content provider full access to the self-service advertising and monitoring facility of the Grid system 104. Recognition software 203 in server 200 is used to identify the subject(s) of each picture that is submitted to the content server 200. Recognition software 203 includes visual, face and word recognition software, and can also include video and audio recognition software. Once a subject, or subjects, of a picture are identified, the recognition software 203 passes the identified subject(s) to the matching software 204. The Grid provider User Interface, recognition software 203 and matching software 204 all use the same classification scheme for populating and searching the library of content 201 . The classification scheme requires that all content, and input from users, be classified into one or more subjects. The matching software 204 searches the library of content 201 for interactive content that matches the identified subject(s). The matched content is then sent to users via user interface 206. User interface 206 includes communications hardware and software that supports communications with multiple users, wherein the typical user is a wireless or wired consumer, purchaser, researcher, or just a person interested in learning more about something they just saw or read.

Still referring to Fig. 2, the content provider interface 207 of the server 200 includes communications software and hardware that supports communications with multiple content providers. The provider interface 207 allows content providers to register with the Grid system 104 and thereafter be allowed to produce links to their own interactive content for storage and subsequent forwarding to interested consumers. After registration, Apps software 202 downloads a Grid-linked provider Ul to the content provider's computer, via provider interface 207. The content provider is then able to access to the collection of self-service files 205 including audio, visual, and audio/video files. The content provider adds a file to be uploaded to the self-service files 205, and then only needs to add content and one or more universal resource locators (URLs) to the file to produce searchable interactive content. In the preferred embodiment, the content provider owns the content that is added to the self-service file. The content provider also hosts the website(s) for the Internet addresses that are contained in the URLs. The content provider then uses the Grid provider Ul to upload the links to the interactive content to the library of content 201 , and subsequently monitor user interactions with the content. Grid system 104 also includes a support server 209 that supports monitoring user interactions with the content and collecting statistics of users' interactions. Support server 209 includes an ACR app module 210 and a metrics module 21 1 . The ACR app module 210 communicates with ACR apps that are downloaded to the consumer/user devices. The ACR app module 210 receives activity reports from the user devices. An activity report includes information regarding the user's interactions with content received from server 200. An activity report can include user's name, ID number, address, IP address, telephone number(s), time of interaction, type of interaction, ID of content, segment of content activated, product purchased, price of product, time of purchase, and other information related to the interaction as permitted by relevant legislation for Data Protection. In the preferred embodiment, the activity reports are automatically sent from the user devices immediately upon user interaction with a link or webpage. The metrics module 21 1 receives the activity report information and prepares the information for viewing by respective content providers. Metrics module 21 1 includes parsing software that process the activity report information and allows for customized viewing of the information. The admin interface 208 provides an interface for a system administrator for management of the system 104.

Fig. 3 is a flow chart showing exemplary steps 300 a user takes to employ the present system 104 in order to quickly receive more information on a subject they are interested in. In step 301 , the user downloads an automatic content recognition (ACR) application (app) to their device. Preferably the user downloads the ACR app from a third party recognition software provider, such as Shazam, Soundhound, Clic2c and/or RedLaser. The Grid system 104 integrates these ACR apps into the system by providing compatibility with the apps, so that searching of the library of content for matches to a submission is enabled. In the preferred embodiment, the user does not need to register with the Grid system. In step 302, the user opens the ACR app and takes a picture of something they are interested in, such as the dress Teri Hatcher is wearing in an edition of the local newspaper. ACR app includes tagging software that is used to tag items in the picture. The tags are subsequently used by recognition software 203 to identify the subject(s) of the picture. In step 303, the ACR app adds tags to the picture. Tags are automatically added by the ACR app, however, the user may also add a tag to a picture, if desired. This is helpful when the picture includes more than one subject, and the user is only interested in one of the subjects shown in the picture. In step 304, the tagged picture is transmitted to the content server 200. Transmission can be wirelessly or via a wired communication medium, depending on the type of device that is being used. In step 305, recognition software 203 is used to identify one or more subjects of the picture. In step 306, matching software is used to find one or more matches to the subjects in the library of content 201 . In step 307, the matched interactive content and software that supplements that ACR app are sent to the user. The supplemental software allows the user to display and interact with the content they receive from the system 104. The supplemental software also includes monitoring functions that allow the system 104 to monitor the user's interactions with the content.

Referring to Fig. 4, an exemplary graphical user interface 400 for the present

Grid system 104 is illustrated. A number of interactive buttons 404 - 407 are provided in the interface 400. Button 404 allows for selective control of acquisition of an image, i.e., allows the user to take a picture. In the preferred embodiment, the picture is automatically submitted to the content server 200 after being taken by the user. Button 405 allows the user to import a picture file from another location, such as from a removable storage device, another digital device, or a location on the Internet. Button 406 allows the user to submit the imported picture file to the Grid system 104. Button 407 is a toggle button that allows the user to start and stop recording a file. The file can be an audio file, a video file, or an audio/video file. A more detailed description of the user recording a file is discussed in conjunction with Fig's 10-1 1 . The primary display area 401 of the interface 400 allows for viewing of multiple primary windows 408 - 410. The primary display area 401 can be used to display: a picture 408 that the user just took or imported; a web page 409 that the user was taken to after clicking on a link received from the content server 200, or other content 410 such as text documents, multi-media documents and a playback interface for an audio, video or audio/video files. Secondary display area 402 provides a display area for interactive content 403 received from the content server 200. The interactive content 403 may include links to websites, video files, audio files, audio/video files, image files, multi-media documents and text documents. When the user clicks on, or other wise activates, one of the links in the secondary display area 402, the link is executed by the ACR app, meaning the path provided by the link is followed, and the associated object (webpage, file or document) is displayed in the primary display area 401 . If the user clicks on a second link in the secondary display area 402, the object associated with that link is displayed in the primary display area 401 in a separate window. If a web page, document or other object shown in the primary display area 401 includes hyperlinks, the user can activate the hyperlinks and other windows will be provided as necessary. Thus, the user is allowed to easily browse through, and interact with, all content and links that they receive from the content server 200. In an alternate embodiment, the user is able to receive a video file (content) that is displayed over the entire display screen of the user's device. Thus, only one display area is viewable. Websites that the user is taken to after clicking on a link from the content server 200 are preferably not hosted by the content server. Rather, each content provider is responsible for the hosting of all websites associated with URLs that the content providers input to the system 104.

Fig. 5 is a flow chart showing exemplary steps 500 content providers can take to employ the Grid system 104. In step 501 , a content provider downloads a Grid provider user interface (Ul) to their computer. The content provider preferably accesses a Grid website via the Internet to download the Grid provider Ul. The Grid website may, or may not, be hosted on the content server 200. In step 502, the content provider registers with the system 104. Registration of content providers involves setting up an account for the content provider. This is an important step in the process and ensures that the content provider is legitimate, and avoids abuse of the account. In step 503, the content provider logs in to the Grid system 104 using the Grid provider Ul. Once logged in, the content provider is provided access to the collection of self service files 205, including audio, video and audio/video files. In step 504, the content provider adds one of the self-service files and adds desired content (e.g. an Ad) and URL(s). In the case where a content provider is an advertiser that has a video clip with sound that they want to publish, the provider adds an audio/video file to the collection 205. The Grid provider Ul provides input fields for the content provider to input the video clip and one or more URLs associated with the video clip. The content provider interface display is discussed further in conjunction with Fig. 7. Associating URLs with advertisements such as the video clip is discussed further in conjunction with Fig. 6. In step 505, after adding the content and URL(s), an entry for the library 201 has been produced. In step 506, the produced content is uploaded to the content server 200 and used to populate the library of content 201 . The interactive content is then searchable and ready to be sent to users as matching content. In step 507, the content provider uses the Grid provider Ul to monitor user interactions with their content. Providers are able to see, depending on the level of data available from the ACR used, when, where and to whom their content was sent. Providers can also monitor the type of interaction users have with their content, such as viewing, requesting more information, and making purchases.

Fig. 6 is a flow chart showing more detailed steps 600 involving a content provider's interactions with the Grid system 104. In step 601 , using the Grid provider user Interface the content provider access the collection of self-service files in the content server 200 and adds an appropriate file for their content. In step 602, the provider adds their content to the file. If the content provider is an advertiser and has a video clip with sound to publish, the provider will add an audio/video file to the collection of self-service files. Upon selection of the file, input fields are displayed to the provider and the provider adds the video clip in the content field. In step 603, the content provider adds one more URLs associated with the content to the input fields for URLs. In step 604, the content provider is allowed to input instructions regarding which URL to activate when the user clicks on the video clip/content. The content provider is allowed to associate different URLs with different segments of the video clip, so that the user is taken to a specified URL depending on when they click on, or otherwise activate, the content. The content provider may also include instructions regarding which URL to provide to a user based on the time of day the library 201 is searched, i.e., when the user submits the picture to the Grid system 104, and/or based on the number of submissions that match their content. The content provider can also include instructions regarding other media files. In step 605, the content provider selects a category or subject that the interactive content will be classified under in the library 201 . Alternatively, the content server 200 may be responsible for classifying the produced content. In step 606, the produced interactive content is uploaded to the content server 200 and the classified content is used to populate the library 201 . The content is then available for searching and forwarding to users of the Grid system 104.

Referring to Fig. 7, an exemplary content provider interface display 700 for the Grid system 104 is illustrated. After selecting a file, such as an audio/video file, from the collection of self-service files 205, content providers are presented with a Create Entry window 701 that allows them to produce links to interactive content, such as an interactive advertisement, for placement of in the library 201 of searchable content. Input field 702 allows the content provider to select or import content, such as an ad in the form of a video clip. After registration with the system 104, the content provider interface display 700 can be customized for the provider wherein, the provider is able to select content from a drop down menu associated with input fields 702 & 703. One or more URLs, or return paths, are input, imported or selected in field 703. The system 104 associates the URL(s) with the content, so that when matched content is sent to a user, the URL(s) is sent with the content. Input field 704 allows the provider to select which category(s) or subject(s) the content will be classified under in the library 201 . The classification subjects that the provider is allowed to choose from, are the same classification subjects used by the ACR app, the recognition software 203 and the matching software 204. If the content provider only has one URL that he is associating with the content, then they can go straight to window 710. However, if the provider has more than one URL that they are associating with the content, then they can go to window 705 and can provide instruction regarding when to present the URLs to the users. More than one URL can be presented together to the user, where the user can select from this menu which link they want to click. By clicking on button 706 the content provider can select how to base distribution of the URLs. Input field 707 allows the provider to base distribution, or presentation, of the URLs on Time. Multiple URLs can be sent with an audio/video content, and the content provider can dictate which URL to activate based on the segment in the video that the user clicks on, or otherwise activates. Thus, presentation of a URL can depend on the time within a video, or segment within the video, that the user interacts with the video. The content provider can also base distribution of URLs on the time of day that the content is matched with a submission. In other words, content that is matched during morning hours can have a first URL that is sent with it, while the same content that is matched during evening hours can have a second URL that is sent with the content. Input field 708 allows the content provider to control URL distribution based on a location of the user. In the preferred embodiment, a cellular telephone with camera is used as the digital device to submit pictures to the system 104. Thus, the location of users can be determined by well-known techniques. Also, user profiles can be collected during registration of the users, and user locations can be looked up in a database of user profiles by the content server 200. The content provider can provide instructions regarding which URLs to present in which locations. Input field 709 allows the content provider to base distribution of the URLs on the number of submissions, or searches, that match the provider's content. Thus, the first 1 ,000 matches for an entry are sent a first URL, and after 1 ,000 matches all subsequent matches for the same entry are sent a second URL. Of course, the above instruction input fields are only exemplary and other instruction input fields could also be provided. For example, the content provider may want to base URL distribution on user profiles when this information is available, and one or more input fields could be provided facilitating such distribution. After inputting or selecting all requirements, the content provider clicks on button 710 and the Grid provider Ul checks to make sure the interactive content complies with all formatting and other system requirements. The Grid provider Ul then uploads the produced content to the content server 200 where it is used to populate the library of content 201 . If the content provider clicks on button 71 1 , they are taken to another interface display (shown in Fig. 9) where the content providers can view metrics regarding user interactions with their content.

Fig. 8 is a flow chart showing exemplary monitoring steps 800 regarding communications between a user's device and the Grid system. In step 801 , monitoring software is downloaded to the user's device when the user receives interactive content from the system 104. In step 802, the monitoring software recognizes when the user receives interactive content from the content server 200, and monitors the content for any user interaction. In step 803, the monitoring software detects an interaction with the content, and sends an activity report to the ACR app module 210. The detected interaction can be the activation of a URL that was sent with the content. The activity report could include information, when available from the ACR app, regarding the user, the URL, the type of interaction, the time of interaction, location of the user, as well as other information. In step 804, the ACR app module 210 forwards the activity report data to the metrics module 21 1 where the metrics for that content provider are updated. In step 805, monitoring software detects another user interaction and sends another activity report to ACR app module 210. The detected interaction this time can be an interaction within a web page, such as the purchase of a product offered for sale on the web page. The activity report in this case could also include product purchased, price of product and other useful information regarding the purchase. In step 806, the ACR app module 210 forwards the activity report data to the metrics module 21 1 where the metrics for that content provider are updated again. The constant updating of the metrics module allows the content providers to view user interactions with their content in real-time.

Fig. 9 illustrates an exemplary content provider interface display 900 that allows content providers to select and view desired metrics. Content providers can arrive at this display 900 by clicking on the Monitor User Interaction Metrics button 71 1 in interface display 700. Button 901 allows the content providers to quickly view a standardized display of metrics. In the preferred embodiment, the standardized display includes a listing of the most recent user activity with the provider's content. Button 902 allows the provider to view metrics regarding their content based on the category or subject of the content. The content provider can view metrics regarding all content (e.g. Ads) in the selected subject by clicking button 903. The content provider can limit presentation of metrics to one or more selected content (Ads) within the subject by clicking button 904. Preferably, button 904 includes a drop down menu that lists all of the provider's content in any selected subject. Lists of content (e.g. Ads) within each subject are stored by the Grid provider Ul, and updated each time the provider produces new content. Window 905 provides display options to the content providers that allow the providers to select which statistical information is displayed. By clicking button 906 the Time of the

interactions will be included in the display of metrics. Button 907 allows the provider to select the Type of interaction(s) they wish to view. Button 908 will allow the provider view user profiles of users that interacted with their content, when this information is available from the ACR app. Button 909 causes the locations of the interacting users to be displayed. And, button 910 allows the provider to view the number of matches that were made with any selected content. Buttons 906 - 910 are only exemplary and more display option buttons can be provided for other metrics. For example, buttons for product purchased and number of purchases could also be provided. Window 91 1 is the display of metrics window wherein interaction metrics are presented for viewing in real-time. An overall column header 912 indicates that the provider selected all Content (e.g. Ads) in the Home

Entertainment category for viewing. The first column header 913 indicates that metrics in that column relate to metrics related to the provider's 1 ^st Item of content (e.g. Ad) in the category. The second column header 914 indicates that metrics in the second column relate to metrics of the provider's 2^nd Item in the category. And, the third column header 915 indicates that metrics in the third column relate to metrics of the provider's 3^rd Item in the category. Row headers 916 & 917 indicate the provider selected Time as a metric they wished to review. Row header 918 indicates the provider also selected Type of interaction as a metric they wished to view. The Type of interaction distinguishes purchases made by users while interacting with the provider's content, from other types of interactions.

Figure 10 illustrates an alternate embodiment wherein the user 100 submits a recorded file to the Grid system 104. In the alternate embodiment the user 100 records an audio, video or audio/video file on their digital device 101 , submits the file to the Grid system 104 and received interactive content regarding a subject of the file. In this example, the user 100 is watching an episode of their favorite TV show, "Desperate Housewives" on a television set 1000. The user's favorite television star, Teri Hatcher, is wearing an outfit, including dress, hat and pocket book, that the user 100 believes would look great on them. The user 100 grabs their digital device 101 , a smart phone is this case, points the camera on the phone toward the TV 1000 and records a short video clip of Ms. Hatcher wearing the outfit. The automatic content recognition (ACR) app in the digital device 101 tags the file, and the file is transmitted to the Internet 103 via a wireless communications channel 106. The file is routed through the Internet 103 and communications channel 107 to the Grid system 104. Just as in the preferred embodiment, the Grid system 104 includes a library 201 of interactive content that is populated by content providers via content provider computers 105. The system 104 searches the library 201 , or provides for third party recognition software to search the library, for content that matches the subject of the file, and sends the matched content to the user 100. In cases where the system 104 identifies the submission, either a picture or a recorded file, as a TV show, the system 104 is able push additional content to the user as the TV show progresses. In the case of a user watching "Desperate Housewives", the user could be sent links relating to actors and their wardrobes as the actors appeared in the show. By dynamically updating the content that is sent to the user, the system 104 anticipates the actions of the user, and becomes a more efficient as a marketing tool. The same hardware and software (shown in Fig. 2) that is used in the preferred embodiment to identify the subject of a picture and match the subject to content in the library 201 , is also used in the alternate embodiment to identify the subject of a file and match the subject to content in the library 201 . Further, since the user 100 is still using the supplemented ACR app to interact with the system 104, the same monitoring functionality is also available to content providers regarding the ability to monitor user interactions with the content in real-time. This alternate embodiment continues the theme of the present invention, which is submitting the subject of a non-interactive source, such as a TV or radio, and providing the user with interactive content relating to the subject.

Fig. 1 1 is a flow chart showing exemplary steps 1 100 in the alternate embodiment. In step 1 101 , the user downloads the automatic content recognition (ACR) application (app) to their device. Preferably the user downloads the ACR app from a third party recognition software provider, such as Shazam, Soundhound, Clic2c and/or RedLaser. The Grid system 104 integrates these ACR apps into the system by providing compatibility with the apps, so that searching of the library of content for matches to a submission is enabled. In step 1 102, the user opens the ACR app and records an audio, video or audio/video file of something they are interested in, such as the outfit Teri Hatcher is wearing on TV. The ACR app includes tagging software that is used to tag items in the file. The tags are

subsequently used by recognition software 203 in content server 200 to identify the subject(s) of the file. In step 1 103, the ACR app adds tags to the file. In step 1 104, the tagged file is transmitted to the content server 200. Transmission can be wirelessly or via a wired communication medium, depending on the type of digital device that is being used. In step 1 105, recognition software 203 is used to identify one or more subjects of the file. In step 1 106, matching software 204 is used to search the library of content 201 and find one or more matches to the subject(s). Alternatively, the system 104 allows for third party software such as Shazam to search the library of content 201 for matches to the subject(s). And, in step 1 107, the matched links to interactive content, along with supplemental software for the ACR app, are sent to the user. Using the supplemented ACR app, the user is able to display and interact with content related to their submitted file. Once again, the user is provided with links to interactive content related to the subject of a non- interactive source. As in the preferred embodiment, the ACR app is supplemented with monitoring software that sends activity reports to the ACR app module 210 upon user interactions with the content. Thus, real-time monitoring of the interactions is available to the content providers.

Fig. 12 is an exemplary content provider interface 1200 that shows an exemplary format for a content provider's entry to the Grid system 104. Entry format window 1201 includes input field 1202 that helps the content provider achieve the proper format for their entry and also ensures that a unique entry is created. Field entry window 1203 includes input fields 1204-1208 that preferably present the provider with drop down menus for selecting among proper codes for the entry.

Input field 1204 is a three letter advertiser code that is allocated automatically to the provided when registering with the Grid system 104. Input filed 1205 is a four letter product code or brand code. Each product code is preferably associated with a product category. Exemplary product categories include Alcohol, Audio-visual (e.g. home entertainment), Beauty and cosmetics, Business (office services, security etc.), Charity, Clothing and footwear, Computers, Education, Employment, Energy, Financial, Food and drink, Gambling and lotteries, Health, Holidays and travel, Household, Industrial and engineering, Leisure and entertainment, Motoring, Noncommercial, Property, Publishing, Retail, Telecommunication, Tobacco, Toys, and Utilities. Input field 1206 is a six digit numerical field that denotes the date of the submission. In the preferred embodiment a day-month-year (DDMMYY) format is used. Input field 1207 is a three digit numerical field that is an assigned submission number for that day. Input field 1208 is a three letter code that dictates what medium is to be used. Exemplary media include Brochure / Catalogue / Cinema / Circular / Direct mail / Leaflet / Magazine / Outdoor / Packaging / Point of sale / Press / Radio / TV (spot) / TV (teleshopping). Exemplary codes for the media are

Bro/Cat/Cin/Cir/DMa/Lea/Mag/Out/Pac/PoS/Pre/Rad/TVS/TVT, respectively. Of course, the above format is only exemplary and the system 104 could allow other formats to be used. For example, the format could include more or less than 19 characters, and fields that are currently numbers could use letters, and fields that are currently numbers could use letters.

Fig. 13 illustrates an alternative embodiment where the user 100 records a file from an object 1300 (for example, by using a camera or microphone on the digital device), such as a television, physical printed material, online displayed material, radio, any audio, visual and/or audio/visual device, or real world object, using an application (such as an ACR application) on a digital device 101 . The application on the digital device 101 extracts, from the file, data to transmit to the Grid System 104. The data may be a fingerprint calculated from the file to assist the Grid System 104 in recognizing the subject of the file, or the data may be a portion of or the entire file. The data is transmitted to the Grid System 104. In one embodiment, the Grid System 104 comprises an identification server 1301 and a content server 1302, and in another embodiment the identification server and content server are part of the same apparatus. In a yet further embodiment, the identification server is under the control of a third party such as the owner of the ACR application. The identification server 1301 uses recognition software to identify the subject of the file and generates an identifier for the subject. To help identify the subject from the data the identification server 1301 utilises content previously uploaded and associated with the subject by content providers 105 through a self-service facility provided by the Grid System. The previously uploaded content may be tagged by content providers with

associated subjects. For example, a user may record the audio from a television advertisement, a fingerprint of the audio may be sent to the identification server 1301 , the identification server 1301 may have previously uploaded content representing the advertisement (such as the entire audio-visual advertisement, the audio track of the advertisement, or fingerprint data about the advertisement), the identification server 1301 may use the previously uploaded content to identify the subject of the file. The identifier may then be used by the content server 1302 to identify appropriate links to interactive content previously uploaded and associated with a subject by the content providers 105 through the self-service facility provided by the Grid System. The content server 1302 may transmit those links to the digital device of the user 100 who may then access corresponding interactive content on their digital device 101 . The present invention has industrial applicability in electronic commerce. Content providers, including advertisers, will be motivated to use the self-service facilities to produce links to interactive content, including Ads, and take advantage of the targeted distribution of content provided by the system. Users, including consumers, are motivated to use the present system so that they can receive interactive content regarding a non-interactive source.

The foregoing description of the embodiments will so fully reveal the general nature of the invention that others can, by applying current knowledge, readily modify and/or adapt for various applications such specific embodiments without departing from the generic concept. For example, the user may take a picture using a digital camera, download the picture to their desktop computer and submit the picture to the Grid system from their desktop computer. Therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the invention. It is to be understood that the phraseology of terminology employed herein is for the purpose of description and not of limitation. furnished blank upon filing

Claims

CLAIMS What is claimed is:

1 . A system that allows a user to take a picture of any printed material, submit the picture to the system, and receive interactive content regarding a subject of the picture, the system comprising: a content server that stores a library of interactive content on a wide range of subjects, and adapted to receive the picture from the user, the content server having recognition software that identifies the subject of the picture, and sends content associated with the subject of the picture back to the user;

the content server having a user interface that facilitates electronic

communications with the user;

the content server having a structured database that stores the library of interactive content;

the content server further having a content provider interface that facilitates electronic communications with content providers;

wherein, the content server provides a self-service facility for the content providers by allowing the content providers to upload content to the content server, wherein matching software within the content server, or a third party software module, matches the uploaded content with pictures that have been submitted to the system, and the system sends matched content to submitters of the pictures.

2. The system of claim 1 , wherein the user takes the picture using a digital device and submits the picture to the content server over a communications network, and,

the digital device includes an automatic content recognition (ACR) application

(app) from a third party, the ACR app communicating with the content server so that the user is able to receive interactive content from the content server, wherein the content server supplements that ACR app with monitoring software that provides for monitoring user interactions with the interactive content.

3. The system of claim 1 , wherein the content provider interface allows for download of a Grid provider User Interface to content provider computers, the Grid provider Ul providing the content providers with access to add to a collection of self- service files, including audio, visual, and audio-video files, that content providers can use to produce links to interactive content, the content provider interface facilitating upload of links to the produced interactive content to the content server.

4. The system of claim 2, further comprising a support server that is coupled to the content server, the support server comprising: an automatic content recognition (ACR) module that communicates with the ACR app, the user interface and the content provider interface to provide tracking information in real-time regarding user interactions with the interactive content; a metrics module that communicates with the ACR module and collects metric data regarding the user interactions and makes the metric data available to the content providers, the metric data relating to number of submissions, location, time, type of interaction or a user profile, depending on the level of data available from the ACR app.

5. The system of claim 3, wherein the produced interactive content includes at least one link including a universal resource locator (URL) or other media files, as listed in (8) that directs the user to a webpage when the user clicks on, or otherwise activates, the link, wherein the content providers instruct the Grid provider Ul which URL to include in the link.

6. The system of claim 2, wherein the interactive content received from the content server includes at least one link that when clicked on, or otherwise activated, by the user connects the user's digital device to a webpage or service selected by the content provider, and, wherein the ACR app automatically contacts the system with interaction information when the user activates the link, and when the user interacts with the webpage.

7. The system of claim 4, wherein the ACR module receives activity reports from the user interface regarding user interactions with the interactive content, and provides the activity reports to the metrics module for parsing, storing and

distribution, and wherein the content provider interface allows for access to the metrics server so that the content providers can view in real-time metrics regarding interactions with their content.

8. The system of claim 1 , wherein the content that is sent to the user includes an audio file, a video file, an audio-visual file, an advertisement, a coupon, a gift card, documentary or editorial information regarding the subject of the picture, a text document, a document with hyperlinks, a multi-media document, a link to a webpage, or other content related to the subject, and further wherein follow-on content is subsequently sent to the user after the content is sent, the follow-on content being pushed to the user and also being related to the subject.

9. The system of claim 3, wherein one of the content providers uses an audio- video file which it has uploaded to the content server to produce interactive content, and the content provider is able to associate different URLs with different segments of the audio-video file, so that if a link associated with the file is activated at the beginning of the file, a person viewing the file will be taken to a first website, however, if the link is activated at the end of the audio-video file the person is taken to a second website.

10. The system of claim 3, wherein content provider interface, the Grid provider Ul, and the collection of files allow the content providers to include instructions with the produced interactive content, wherein the instructions provide for different URLs being sent with matched content, based on a user profile, a time of day, a location, or a number of submissions.

1 1 . A method that allows for a user to take a picture of any visual material, submit the picture to a content server that has a library of interactive content, and receive interactive content regarding a subject of the picture, the method comprising the steps of:

providing a user interface to the content server that facilitates submission of pictures to the server and transmission of content to the user;

providing a content provider interface to the content server that allows content providers to produce interactive content and populate the library with the content by uploading the interactive content to the content server;

receiving a picture file by the content server via the user interface;

employing recognition software within the content server to determine the subject of the picture; matching the picture with interactive content stored in the library based on the determined subject; and,

sending the matched interactive content to the user, wherein the matched interactive content provides the user with additional information regarding the subject of the picture;

wherein, the content server provides a self-service facility for the content providers by providing a collection of self-service files uploaded by content owners that allow the providers to produce the interactive content and a classification scheme that facilitates populating of the library with the produced content.

12. The method of claim 1 1 , wherein the user takes the picture using a digital device and the picture is submitted to the content server over a communications network, and

the step of providing a user interface, further comprises downloading an automatic content recognition (ACR) application (app) to the user's digital device, the ACR app allowing the user to receive links to the interactive content from the content server, and allowing monitoring of user interactions with the content.

13. The method of claim 1 1 , wherein the step of providing a content provider interface further comprises, allowing for download of a Grid provider user Interface to content provider computers, the Grid provider Ul providing the content providers with access to the collection of self-service files the providers have uploaded within the content server, including audio, visual, and audio-video files, that content providers can use to produce the interactive content.

14. The method of claim 12, further comprising the step of:

coupling a support server to the content server, the support server having: an automatic content recognition (ACR) module that communicates with the ACR app, the user interface and the content provider interface to provide tracking information in real-time regarding user interactions with the interactive content; and, a metrics module that communicates with the ACR module and collects metric data regarding the user interactions and makes the metric data available to the content providers, the metric data relating to number of submissions, location, time, type of interaction or a user profile, depending on the data made available by the ACR app.

15. The method of claim 13, wherein the step of providing a content provider interface further comprises including in the produced interactive content at least link that includes a universal resource locator (URL) that directs the user to a webpage when the user clicks on, or otherwise activates, the link, wherein the content providers instruct the Grid provider Ul which URL to include in the link.

16. The method of claim 12, further comprising the steps of:

including in the interactive content received from the content server at least one link that when clicked on, or otherwise activated, by the user connects the user's digital device to a webpage selected by the content provider, and,

automatically contacting the system with interaction information when the user activates the link, and when the user interacts with the webpage, wherein the ACR app facilitates contacting the system.

17. The method of claim 14, further comprising the steps of: receiving, by the ACR module, activity reports from the user interface regarding user interactions with the interactive content;

providing the activity reports to the metrics module for parsing, storing and distribution; and, allowing the content provider interface to access the metrics server so that the content providers can view in real-time metrics regarding interactions with their content.

18. The method of claim 1 1 , wherein the content that is sent to the user includes an audio file, a video file, an audio-visual file, an advertisement, a coupon, a gift card, documentary or editorial information regarding the subject of the picture, a text document, a document with hyperlinks, a multi-media document, a link to a webpage, or other content related to the subject, and further wherein follow-on content is subsequently sent to the user after the content is sent, the follow-on content being pushed to the user and also being related to the subject.

19. The method of claim 13, further comprising the step of: using, by one of the content providers, an audio-video file from the self- service files to produce an interactive content, wherein the content provider is able to associate different URLs with different segments of the audio-video file, so that if a link associated with the file is activated at the beginning of the file, a person viewing the file will be taken to a first website, however, if the link is activated at the end of the audio-video file the person is taken to a second website.

20. The method of claim 13, wherein the Grid provider Ul and the collection of files allow the content providers to include instructions with the produced interactive content, wherein the instructions provide for different URLs being sent with matched content, based on a user profile, a time of day, a location, or a number of

submissions.

21 . The method of claim 12, wherein the user's digital device includes a display screen, and further comprising the step of:

displaying the received content in a second portion of the screen, wherein the picture, or other data, is displayed in a first portion of the screen, wherein upon receipt of special content, the special content is displayed over the entire display screen including the first and second portions.

22. The system of claim 2, wherein the user's digital device includes a display screen, and the received content is displayed in a second portion of the screen, wherein the picture, or other data, is displayed in a first portion of the screen.

23. A system that allows a user to record a file from any file source wherein, the file source outputs audio, video, or audio and video data, submit the file to the system, and receive interactive content regarding a subject of the file, the system comprising:

a content server that stores a library of content on a wide range of subjects, the server adapted to receive the file from the user, the content server having recognition software that identifies the subject of the file, and sends interactive content associated with the subject of the file back to the user;

the content server having a user interface that facilitates electronic

communications with the user; the content server having a structured database that stores the library of content, wherein matching software used to match the subject of the file to content stored in the library;

wherein, the content server provides a self-service facility for the content providers to upload content to the content server, and an automated searching facility for the user by allowing the user to search through the library of content.

24. The system of claim 23, wherein the user records the file using a digital device and submits the file to the content server over a communications network, and,

the digital device having an automatic content recognition (ACR) application (app) that communicates with the content server and provides for submitting files, receiving matched interactive content, displaying the content, and displaying objects, including web pages, audio files, audio/video files and multi-media documents, linked to the content.

25. A method that allows a user to record a file from any file source wherein, the file source outputs audio, video, or audio and video data, submit the file to a content server with a library of content, and receive interactive content regarding a subject of the file, comprising the steps of:

using a digital device with a camera and/or microphone to record the file; sending the file to the content server wherein, the library of content is populated by content providers that upload interactive content to the server;

using recognition software within the server to identify a subject of the file; searching the library of content for interactive content that matches the subject of the file;

sending the matched interactive content to the digital device;

wherein, the content server provides a self-service facility for the content providers to upload content to the content server, and an automated searching facility for the user by allowing the user to search through the library of content and receive interactive content related to the submitted file.

26. The method of claim 25, wherein the digital device includes an automatic content recognition (AC ) application (app) that communicates with the content server and facilitates submitting files, receiving matched interactive content, displaying the content, and displaying objects, including web pages, audio files, audio/video files and multi-media documents, linked to the content.

27. A system that allows a user to record an audio, visual or audio/visual file, submit data obtained from the file to the system, and receive interactive content regarding a subject of the file, the system comprising:

an identification server adapted to receive the data from the user, the identification server having recognition software that identifies the subject of the file from the data and generates an identifier;

a content server that stores a library of links to interactive content associated with one or more of a plurality of subjects, and adapted to receive the identifier, the content server using the identifier to find and send links to interactive content associated with the subject back to the user;

the content server having a structured database that stores the library of links to interactive content;

wherein, the system provides a self-service facility for the content providers by allowing the content providers to upload content and associated subjects to the identification server and to upload links to interactive content to the content server.

28. A method that allows a user to record an audio, visual or audio/visual file, submit data obtained from the file to an identification server of a system, and receive a link to interactive content regarding a subject of the file from a content server of the system, comprising the steps of:

using a digital device with a camera and/or microphone to record the file; sending data from the file to an identification server;

using recognition software within the identification server to identify a subject of the file from the data and generate an identifier;

a content server searching the library of content for links to interactive content that matches the subject of the file using the identifier; sending a link to the matched interactive content to the digital device;