EP1197020B2

EP1197020B2 - Electronic music and programme storage, comprising the recognition of programme segments, such as recorded musical performances and system for the management and playback of these programme segments

Info

Publication number: EP1197020B2
Application number: EP00920069A
Authority: EP
Inventors: James D. Logon; Richard S. Goldhor
Original assignee: Gotuit Media Corp
Current assignee: Gotuit Media Corp
Priority date: 1999-03-29
Filing date: 2000-03-29
Publication date: 2011-04-13
Anticipated expiration: 2020-03-29
Also published as: EP1197020B1; WO2000058940A2; AU4066300A; DE60037119T3; DE60037119T2; EP1197020A2; WO2000058940A3; DE60037119D1

Abstract

A system for selecting, erasing or reproducing program recordings using marking and descriptive data which is transmitted to a client location from a remote processing location. A database of identification signals specifying the characteristics of a known programming is maintained at a remote processing location. In a first embodiment, selected identification signals are downloaded from the database to the client location and are used by a processor at the client location to identify desired programming within a locally stored collection of previously received broadcast programming signals. In a second arrangement, locally stored programming signals are processed to extract identification data which is uploaded from the client location to the remote processing location for comparison to the database, and information describing the content of the matching programs is returned to the client location for use as a program guide, facilitating the selection, permanent storage, or playback of desired program records and/or the erasure of undesired programming. To conserve local storage space, identified program records may be uploaded and stored at the remote processing location, or shared program records in a central library may be made available for remote playback after the identity of equivalent locally stored programming is confirmed.

Description

Technical Field

This invention relates to systems for storing and reproducing recorded music and programming materials and more particularly to program storage and playback systems in which recorded program content and descriptions are transferred between a client location and a remote processing location.

Background Art

The present invention represents an extension to the systems and methods described in U.S. Patent 5,892,536 issued to James D. Logan, Richard Goldhor and Daniel Goessling on April 6, 1999 and in International Publication WO 98/31113 published on July 16, 1998.
International Publication WO 98/31113 describes an arrangement of the type illustrated in Fig. 1 of the attached drawings which enables a user at a client location to identify and selectively play back or erase program selections, such as individual songs, using program guide information which is downloaded from a remote processing location.
European patent application EP-A-0 898 278 discloses a transmitting and receiving system wherein a data file produced by a user can be uploaded into a server and the updated data file is downloaded to another user. The transmitting and receiving system provides a place where general users can lay works thereof open.

Summary of the Present Invention

As the listener accumulates a large number of previously recorded programs or program segments (e.g. songs) which he or she desires to retain, the present invention provides methods for moving those recordings to a central server to free space on the local storage unit, and to allow the user to access the stored material from other players. To this end, the present invention as shown in Fig. 2 of the drawings further includes a mechanism, seen at 123, for identifying programs as shown in Fig. 1, and further for transmitting. programs and programs segments to be added to the library file 125 maintained at the remote server.
As further contemplated by the present invention, to avoid the need to upload programs from local storage, the identification mechanism is employed to verify that the programs are stored locally, and to communicate that fact to the remote processor, which maintains an accounting file which specifies which users have the right to retrieve and play back which programs from the shared library.
These and other features and advantages of the present invention will be made more apparent by considering the following details description which is presented in connection with the attached drawings.

Brief Description of the Drawings

Fig. 1 is a block diagram of a prior art client-side program storage and playback device interconnected via a communications pathway to a remote server which recognizes snippets of programs received from the client-side unit and which returns descriptions of matching program segments, such as individual songs, to the play back unit; and
Fig. 2 is a block diagram of program storage and playback system including a mechanism for storing program content on behalf of the user on a central remote shared server.

Description of the Best Mode

The prior art arrangement shown in Fig. 1 of the drawings allows a listener or viewer to enjoy selected, previously broadcast radio or television programs or program segments at a later when it is more convenient or desirable. In particular, previously broadcast musical programming segments (here referred to as "songs") can be easily identified and replayed.
The basic mechanism employed is shown in Fig. 1 of the drawings and consists of a client-side recorder/player, shown at the left of the vertical dashed line 101, and a song identification server shown at the right of the line 101.
The recorder/player consists of a broadcast receiver 103 coupled to an antenna 105 for receiving, demodulating and digitizing broadcast signals and for recording those signals on a substantially continuous manner in a local storage unit 107. On the client-side, a "snippet extractor" \109 sends brief digitized segments, here called "snippets," to the recognition engine 111 at the server.
The recognition engine 111 compares each snippet with a database 113 containing prerecorded programming, such as popular songs. When an incoming snippet from the client recorder/player matches one of the items in the database 113, information describing the matching item is returned to the client side and stored as a record in the stored content guide seem at 115. The transmitted information includes data specifying the time duration between the beginning of the identified snippet and the beginning of the program item (e.g. song) from which the snippet was taken, the time duration between the beginning of the snippet and the end of the program item, as well as descriptive information about the program item (e.g., song title, performer, composer, album name, date performed, etc.). Using the information thus accumulated, the user of the player recorder can review listings of songs that are available in the local song storage unit 107, and play back any song or other program item listed as indicated at 121.
The broadcast receiver 103 may be set by the user to continuously record the broadcast from a preselected a radio station, or may be programmed to switch to different frequencies at different times to record different selected programs from different stations. The incoming signal may be derived from an AM or FM radio broadcast, or from the audio portion of television programm-ing. The principles employed in the arrangement shown in Fig. 1, and in the present invention, are applicable to television programming as well, and may be used to store, catalog and play back television programs and segments of television programs.
A variety of different program extraction and recognition mechanisms may be used to implement the arrangement of Fig. 1 and the invention. See, for example, U.S. Patent 5,577,249 entitled "Method for finding a reference token sequence in an original token string within a database of token strings using appended non-contiguous substrings"; U.S. Patent 4,918,730 entitled "Process and circuit arrangement for the automatic recognition of signal sequences," U.S. Patent 4,739,398 entitled "Method, apparatus and system for recognizing broadcast segments," and U.S. Patent 4,697,209 entitled "Methods and apparatus for automatically identifying programs viewed or recorded."
The signature database 113 and recognition engine 111 preferably takes the form of a shared system to which multiple client-side recorder players may be connected via a suitable digital communications pathway such as the Internet or a direct modem connection via the dialup telephone system. The broadcast receiver 103 preferably includes analog-to-digital converter and a digital compression mechanism to conserve space on the local storage unit 107 and to reduce the size of each snippet sent to the server.
The selection and playback mechanism 121 preferably includes means for displaying a listing of the available programs and program segments stored in local storage unit 107, means for searching the information in the stored content guide 115, and means for selectively playing and erasing selected items in local storage 107 that are identified in the stored content guide information at 115. All of these functions may be performed, if desired, by a suitably programmed personal computer equipped with a TV-Radio tuner card, such as the Hauppauge WinCast/TV-Radio card, and utilizing the PC's local hard disk to provide both the local song storage and storage for the stored program and content guide. An Internet connection from the client PC to a remote server may be used to upload snippets and download program and program segment specification data either continuously or on a batch basis. In the batch mode, the snippet extractor 109 may scan pre-recorded program segments in the store 107 and pass them to the recognition engine 111 for processing at "off hours" when the added computational and communications burden placed on both the client and server side apparatus may be more efficiently handled.
If desired, snippets which are transmitted to the server but not recognized, and which hence represent programming which cannot be automatically cataloged, may be automatically deleted from the storage unit 107 after a predetermined time, whereas recognized program segments may be retained until there disposition is specified by the user.
As the listener accumulates a large number of previously recorded programs or program segments (e.g. songs) which he or she desires to retain, it would be desirable to move those recordings to a central server to free space on the local storage unit, and to allow the user to access the stored material from other players. To this end, the present invention as shown in Fig. 2 of the drawings further includes a mechanism, seen at 123, for identifying and transmitting programs and programs segments to be added to the library file 125 maintained at the remote server. As indicated at 127 in Fig. 2, the program material in the remote library file 128 may be retrieved for playback and returned local storage when desired. Those elements seen in Fig. 2 which provide the same functions as like units shown in Fig. 1 are identified by the same reference numbers.
Multiple users may share the library file 125, with only a single copy of each program segment actually being stored. When a client station signals its intent to store a given program or program segment which has been stored in its local storage unit 115, and that program or program segment is already stored in the shared library file 125 as determined by a server-side account manager routine indicated at 128, rather than actually transferring a copy from the client to the server, the copy at the client side is simply erased and an accounting entry is stored in the account file 129 to indicate that a "virtual transfer" of the file has been made. In this way, the copyright on the broadcast program material is protected against making any copy beyond the single copy for listener's personal use. User's who have not actually first created their own copy in local storage on the client recorder/playback unit cannot obtain an accounting "credit" which will entitle them to download the library file copy.
To insure that only one copy can be used, the system could "lock" the original copy at the time the transfer or virtual transfer was made, making the original copy inoperable even though it is still resident (not yet erased) on the client player. The lock could be opened later after a secure message was received from the server indicating that the accounting credit was being eliminated in the account file 129. In this way, the owner of a recorded program segment can play that segment on different players at different times, while the system insures that only one operative at any one time.
Each song in the library file 128 would thus be available only to authenticated individuals who had earlier uploaded the song to the server. The master copy on the server would remain operable to be downloaded to other individuals A server that would download (or unlock if it was already there) a copy to a user's second PC after verifying that the copy on the first PC was locked thus ensuring one copy per user.
Note also that a user may purchase the right to play and locally record a program segment downloaded from the server. The system that would register record/CD purchasers and allow them to download temporary copies of their purchased material to a remote site following appropriate identification.
Furthermore, users might be able to exchange or sell their personal copies of songs. To this end, the server could manage the sale or auction of previously purchased "virtual copies".
In order to reduce the cost of the service provided by the server, and to compensate the copyright owners of the program material, advertisements could be added between programs or program segments (songs).
In addition to, or as an alternative to, the automatic snippet recognition mechanism discussed above, There are several less elegant, but nonetheless practical methods to identify and mark the boundaries of desired program segments stored in the local storage unit 107. The recorder/playback unit could include means for manually marking the beginning and ending of a desired program segment which the user desires to save in his or her "virtual jukebox. Intelligent fast forward and backward buttons, speech speedup software (with pitch control so one could listen to the music in fast-time) to get to the end quickly, set-time jumps back and forth, etc. could be used to facilitate the markup process. These techniques might work particularly well in the car given that the listener has the time to do the required "work".
A music/talk recognition mechanism may be used to delimit the song. This approach rely on an algorithm or circuit that distinguish music from the spoken word (in something akin to "audio scene-change" in video technology). For example, the arrangement described in U.S. Patent 4,542,525 entitled "Method and apparatus for classifying audio signals" processes an audio signal and derives either a speech recognition signal, a music recognition signal or an indication of an unidentifiable signal.
Another implementation employs algorithms to separate two songs. That is, the system would be able to distinguish the beginning of one song and the end of another-songA/songB recognition. These transition points would then be used to help delimit songs within a stored audio stream. This technique would work in conjunction with a music/talk recognition system that separates the talk from the music would be more important than separating the occasional pair of songs that have no talk between them.
These delimitation techniques would then be applied to the audio stored in a time-shifted radio system. The talk or ads between songs could be eliminated either automatically or through user actions. With the bookmarks separating songs, the listener would be able to use an input device (such as a push button or voice-recognized spoken commands) to quickly surf from song to song. The user could then key in, or dictate using voice recognition, descriptive information about any desired song to be saved for future playback, the descriptive information being placed in the stored content guide 115.
Another form of automatic bookmarking would involve" talk radio." In this environment the system would offer another form of recognition-speaker identification. For talk radio this would allow listeners to jump from segment to segment as new speakers joined in the conversation. Each time a new voice was identified, a bookmark would be planted.
Specific word or phrase recognition would also be used to identify segments. For instance, the traffic report each day might start with the same phrase which could be recognized with standard speech recognition technology. The system would place intelligent bookmarks (intelligent in that they related to a known topic) at these identified locations.
Entire talk shows or news broadcasts could be translated to text via speech recognition. Listeners could use the voice input devices in their car PCs to request topics to hear about. These topics would be selected based on word matches.
Finally, bookmarks could be created that merely related to time. Thus, a listener might surf through a time-shifted block of audio and one of the bookmarks might be the audio corresponding the top of the hour or the breaks at every quarter hour.
In another implementation of a delimiting system, computer readable information, possibly in the form of RDS information identifying the song, performers, etc., might accompany the broadcast of a song. In the case of an Internet broadcast, this might include computer readable data such as the name of the song, performer, etc. If these tags were at a known and consistent location vis a vis the start of the song, this would allow for accurate delimiting of the song. If it was inconsistently placed, but generally located near the beginning of a song, then a song could be excised out but it would have some extraneous material associated with it. A system could be devised that excised out a 5 minute block from a buffer surrounding such an identifier. The user could then manually edit the extraneous material.
Another implementation of time-shifted radio listening would involve a multi-tuner broadcast receiver 103. This system would continuously try to delimit songs on multiple channels at once using our original song recognition algorithms (on the client or server machines), or the ideas of music/talk recognition or songA/songB recognition. Multiple tuners could even be useful in the case of manual markups as this can probably be done faster than real time. One implementation of this would use high end equipment that could digitize all the channels in specific spectrum range with a single circuit.
In the operation of a time-shifted radio system, with limited tuners and limited disk storage, the user in general has three playing options:

1. To play a song out of a stored jukebox (presumably stored long ago)
2. To play a recently stored song (involving a short time-shift)
3. To play a song that was being broadcast live at the moment of play.

The Channel Changer implementation is optimized for the latter two options-options which would be most likely in systems with limited memory, or ones with larger memory but little content yet stored. In these cases the listener is more dependent on what is being broadcast at the present than on what is stored on disk from earlier recordings. As a result, a system that finds the greatest number of good songs quickly will have additional utility.
There are several aspects of the Channel Changer system which attempt to emulate the methods that people use as they search for good songs by switching from channel to channel in a car.
In this implementation, song recognition technology (of any of the types described above) is combined with multiple radio tuners, each with its own buffer, and channel-changing algorithms that would be used to intelligently tune the radio tuners to the optimal set of stations. (One of the tuners would be the "playing tuner", while the others would be "searching tuners".) This system will allow a listener to quickly surf multiple radio channels and "pull down" and store the greatest number of desired songs in the shortest time (compared to a single tuner system or a multi-tuner system without channel-changing algorithms).
The system would consist of a database of song fingerprints and the requisite recognition software. The song fingerprints would be as close to the front of the song as possible (not too close or the DJ might chop it off) so that song identification could happen as soon as possible once the song started playing.
Each searching tuner would have a buffer available to it (which could be as short as the distance from the beginning of a song to the fingerprint plus processing time) to capture the audio before reaching a given song's fingerprint. Once the fingerprint was found and identified, the song would be rated on a "desirability" scale. The audio before the buffer before the fingerprint would be combined with the rest of the song. The next step would be dependent on which playing option was in effect:

1. Under this option, the song would be saved in the jukebox.. If there is not enough unused memory to save the song, the system would compare the new song's rating to that of the song in the jukebox that had the lowest rating. If the new song has a higher rating, it would replace the existing song having the lowest rating. The process would continue over time over multiple tuners, gradually lifting the average rating of songs in memory.
2. Under this option, the song would be saved in a short term buffer and queued up for playback. Again, if the queue was filed, the new song would have to be better than the worst song in the queue to find a space.
3. Under this option (which uses minimum memory), the playing tuner would switch over to the new song as soon as it was being broadcast. There would not necessarily be storage of the whole song. The switch to the new song would be made with some consideration being given to how much better the new song is, and how close the old song is to being over.

This search algorithm to find better stations to tune to using multiple tuners would have several steps. The first would be to rank stations by the probability of finding a desired song. This list could be fluid and change depending on the number of successful searches completed on each station over a recent period of time.
Then, using a predictive algorithm which would keep track of which channels had just played a song and which were in the middle of a song or an ad, the system would predict the probability of finding a desired song in the immediate future on any one given station.
The station ranking and the "song-immediacy probability" would both be assessed to decide which station to send the next available tuner to. There the tuner would wait for the next song to come on and be identified. If would then assess the desirability of the new song and under playing options

1. see if the song warranted space in the jukebox,
2. buffer the song and put it in the playing queue
3. switch the playing tuner over to the newly found song.

(Under scenario 3, the user might want the flexibility to use fingerprints that are deeper into the song. Deeper fingerprints would allow the search tuner to identify a song even if it was "found" while in the middle of its playback. This would allow especially desirable songs to be listened to in part. Ideally however, the system would have a separate tuner for each worthwhile station. This would eliminate having to jump from station to station and deal with partial songs.)
The recorder/playback unit can be programmed to develop song desirability ratings automatically. The rating system could learn from the listener:

1. It could watch to see if, and when, a listener skipped out of a song as it was being played under any of the three listening options.
2. It could monitor if, under option #3, the user un-did a swap presented by the system.
3. The system could also keep track of over how long a time, and how many times, each title had been listened to and deduce a decay rate in the desirability in that song due to the multiple exposures. The system could learn what the listeners typical desired decay rate was.
4. Also, the user could have a simple rating interface whereby he or she could consciously rate songs so the system would know how to rate them later. The rating system could be an ongoing process so that a user could help the system understand over time at what rate the listener might be growing tired of a new system. This could be a simple "thumbs down" button that might depress the songs rating in the personal jukebox so that the song would not be played as often.

In addition, a self-reflective mechanism could be employed. That is, if the user expressed a preference for the Beatles tune #1, the system then knows which songs are most like that and puts the others in the desirability scale accordingly.
For a car-PC system, the buttons on the standard radio could be used for any of the skipping and approval rating functions mentioned above.
An interesting and potentially valuable use for this information would be to provide the song ratings back to the broadcasters or record companies. This information could be uploaded when the system "docks" to receive more song fingerprints if that system is employed, or a separate communications step may be used to transmit this information.
With the vast amount of music in existence and being played over the airwaves, a major problem is figuring out what an individual listener may like-and then using this information to listen to the songs have a higher probability of being desirable. Solving this problem is a three step process. First the listener must get exposed to wide range of new material. Radio broadcast (over the airwaves or Internet) is perfect for this. It pushes at listeners randomized playlists of songs grouped around certain musical interest groups.
The second step is "documenting" the listener's tastes while listening. And the third step is making the song available for replay. This can be done by merely waiting to hear it again on the radio, buying the record, or using the present system to snip it out for future listening. The listener can document his or her preferences and save the song all at the same time.
The car is the ideal environment to use this system as the listener is easily able to hit a button or use a verbal command to rate or store a desirable song when it is being played.
Once a personal jukebox has been developed, playlist software may be utilized. This enables automatic play of either randomized or listener-selected songs in much the same way a CD player does. When in user-controlled mode, the user would be able to work from an audio menu which would announce the group and/or song before playing. The listener could then surf through the jukebox while driving.
In random or playlist mode, the system would adapt and use some of the playlist software used by radio stations to construct the random playlists. This software would continuously scan new songs that had been delimited by the system and offer them as new material to the listener. An audible signal would announce that this was a new song. The user could command the system to discard the song and not play it again or rate the song during the first or a subsequent play which would then allow it a place in the jukebox. The playlist generator would attempt to optimize the presentation of new material to a listener based on past listening habits, surfing actions, and explicit expressions of interest in certain types of music.
Another use for song identification with a stored audio system would be to purchase the song being played much in the same way that the listener can now while listening to an Internet broadcast. Other interactive features would be to allow the listener to request information about the band, etc.
Another use for song recognition technology (whether on the client or server side) would be to help listeners identify a specific song that they can't remember-a computerized version of "name that tune". For instance, a user might be able to sing, or perform with an instrument, a few notes of a song. The system would come back with a match or a list of songs that may match the user's attempt to sing the song. The user could then play some of the songs on the "possible matches" list or may recognize the song by the name. In any case, the user might go on from there to buy the song that matched his or her "performed" rendition or cite it as a song for which it would desirable to capture with the song recognition system when played over the radio again.
It is to be understood that the specific arrangements which have been described are merely illustrative applications of the principles of the invention.

Claims

A method for selectively reproducing locally, stored programming signals comprising, in combination, the steps of:
storing at a client location a programming segment representative of a recorded snippet of broadcasted musical performance,

storing, at a remote processing location, a library of programming segments and content descriptions for those stored programming segments,

at said client location, employing processing means to derive identification data from the recorded snippet of broadcasted musical performance,

transmitting said identification data from said client location to a remote processing location,

at said remote processing location, using a recognition engine to compare the transmitted identification data with said library of programming segments to identify if any of the program segments in the library match the recorded snippet,

if there is a match, transmitting from said remote processing location to said client location the content descriptions for the matching program segments,

at said client location, presenting the transmitted content descriptions to a user to facilitate the selection and retrieval of particular ones of said matching program segments,

at said remote processing location, accepting a retrieval request from said client location specifying one or more of the particular ones of the matching program segments, and

responding to said request by transmitting to said client location the content of said one or more particular ones of said matching program segments.
The method set forth in claim 1 wherein at least some of said content descriptions specify one or more attributes of the corresponding recorded musical performance from the group of attributes consisting of the title, performer, composer, and date of the corresponding recorded musical performance.
The method set forth in claim 1 wherein said step of storing at the client location comprises receiving and recording broadcasted programming signals.
The method set forth in claim 3 wherein said library of programming segments is derived from said broadcasted programming signals received at said remote processing location concurrently with the reception and recording of said broadcast programming signals at said client location, and said content descriptions transmitted to said client location from said remote processing location are used at said client location to facilitate the selective time-shifted reproduction of said broadcast programming signals.
The method as set forth in 1 where said content descriptions transmitted from said remote processing location to said client location include information specifying the beginning and ending time of each said matching program segments.
The method set forth in claim 1 further comprising the steps of uploading a copy of a program segment stored locally at said client location to said remote processing location and storing the uploaded copy in said stored library for later retrieval from said remote processing location.
The method set forth in claim 1 further including the steps of posting an entry in an accounting file upon the transmittal of said identification data to said remote processing location, subsequently transmitting a playback request identifying said client location and identifying a requested program segment, and authorizing the transmittal of said requested program segment if said accounting file contains data indicating the identification data for said requested program segment was previously transmitted from said client location.
The method set forth in claim 7 further including the step of disabling the playback of a local copy of an program segment when the remote playback of said program segment is authorized at said remote processing location.
The method set forth in claim 7 further including the step of disabling the transmittal of said requested program segment from said remote processing location when the playback of the copy of said requested program segment locally stored at said client location is enabled.