IL104725A - System for exchanging sections of video background with virutal images - Google Patents

Description

ΊΝ'ΐη ΌΊΐα y n ,rmy>-> rn»i*T» mmon oy SYSTEM, METHOD AND APPARATUS FOR EXCHANGING SECTIONS OF VIDEO BACKGROUND WITH VIRTUAL IMAGES A SYSTEM, METHOD AND APPARATUS FOR EXCHANGING SECTIONS OF VIDEO BACKGROUND WITH VIRTUAL IMAGES The present invention relates to a system and method for detecting, identifying and scaling in a video frame, suitable distinct targets and areas and inserting into these areas virtual images stored in the memory of the system, so that all objects or shadows in front of the distinct areas blocking portions thereof from view will be seen in a video transmission as being in front of and blocking the same portions of the areas containing virtual images. A particular feature of the invention is to operate the system in real time. The invention also provides apparatus for operating the system.

BACKGROUND OF THE INVENTION International sports events or other spectacles generally draw the interest and attention of spectators in many countries. For example, the Olympics, Superbowl, World Cup, major basketball and soccer games, auto races etc. fit into this category. Such events are generally broadcast live by video to a large international audience. The locale in which these events take place, such as stadiums or courts, provide advertising space all around in the form of signs, posters or other displays on fences and billboards, and in fact on any unoccupied space suitably located, including sections of the playing field.

Due to the nature of the displays, which are mostly in the form of printed matter, they are not changed too frequently and remain at least for a day, or a series or a whole season, and are directed mostly to local audiences. In cases where two teams from different countries play each other, the advertisements are usually arranged so that one side of the stadium contains advertisements directed to audiences in one country, while the other side has advertisements directed to the spectators in the other country. The video cameras in these instances film the event from opposite sides of the stadium for their respective audiences. This of course limits the angle from which the events can be seen in either of the countries represented in the game. Another limitation to present methods of advertising is the stringent safety requirements for positioning the billboards, so as not to interfere with the game nor pose a danger to the players. The displays must not be too close to the actual field of action, so as not to distract the players. A most serious drawback of the present system for advertising at major world sports events is the fact that although the event is televised live throughout the world, the actual physical advertisements in the stadium, because of their broad international exposure, can only cater to products having a world market. Local advertisers can only make use of such world-class televised events by locally superimposing messages on the TV screen, or by interrupting the real time of the event.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a system and method for video transmission of active events, for example sports events, having in the background physical images in designated targets, wherein said physical images are electronically exchanged with preselected virtual images, so that objects or shadows actually blocking portions of the physical images will be seen by viewers as blocking the same portions of the virtual images, and the motion of players or a ball blocking the physical image will block corresponding regions of the exchanged virtual image, so that the exchanged electronic image will remain in the background of the event, exactly as the original image.

Since the same physical image may be captured by a plurality of TV cameras deployed in various locations around the court, and each camera usually has a continuous zoom lens, the system is able to detect and identify a certain physical target in all possible spacial orientations and magnifications of said target. The system is also capable of unequivocally identifying the scale and perspective of said physical target and normalizing the implanted virtual image into the same perspective .

Another object of the invention is to provide a system and method of implanting in video transmission, virtual images in predetermined "free" background areas generally unsuitable for displaying physical signs, like the sports court itself.

Another object of the invention is to perform the implantation described above on a succession of video frames .

Yet another object of the invention is to provide the above system and method for electronic exchange or planting of virtual images in real time.

A further object of the invention is to provide a system and method for video broadcasting the same event to different populations of viewers in real time, with different electronic messages substituted in the spaces occupied by physical displays.

Still another object of the invention is to provide a system and method for utilization of available space in a stadium unused by physical displays for the purpose of advertising by planting therein electronic virtual images during real time broadcasting of an event taking place in a stadium.

Still a further object of the invention is to provide apparatus for use in video transmission for exchanging physical images with virtual images or planting virtual images in unused background areas during an event in real time video transmission, without disturbing the actual transmission of the event.

In accordance with this invention, there is provided a system and method for broadcasting active events being filmed by a TV camera, wherein virtual images are electronically substituted in or superimposed on targets selected from physical displays and preselected background regions, comprising: an electronic data bank of event locales and targets therein; a memory unit for storing digitized virtual images for substitution in said targets; means for grabbing and digitizing video frames; means for target searching in digitized video frames and for detecting targets therein; means for verifying and identifying said targets; means for comparing the detected targets with corresponding targets in the data bank; means for scaling and identifying the perspective of the original target and transforming the virtual substitute image into the same scale and perspective; means for real-time video tracking of a detected target throughout a succession of frames; means for distinguishing between non-background objects and shadows that block portions of said detected targets ; means for electronically transferring said objects and shadows from said original video frame to said substituted frame; means for inserting said electronically transformed virtual image into said video frame substituting the original image in the target; means for receiving and storing virtual images and generating a virtual images data bank; means for generating a locale data bank either prior or during an event (a learning system); and video signal input-output means.

For this purpose the system uses a special algorithm for the detection and identification of targets using one or more of the following attributes: - geometry - such as the physical configuration of billboards (rectangular shape) as seen from different angles and magnifications; - texture of slogans and graphics - such as for example in posters; - character recognition; - field or court lines - which serve as references for designating free court areas; - standard objects that have typical shape and texture - such as post, backboard, basket and/or a player's shirt; - colour; and - objects and shadows temporarily blocking portions target at any capturing angle and range and in any zoom state, and preferably in real time. It should identify in any frame a relatively large number of targets (up to 20 targets or more in an extreme case).

Blocking objects and - s adows. _are distinguished""from the background image by means of: comparing the detected target (partially blocked) with the same target stored in the system's data bank. The image difference between the two is the image of hidden surfaces which forms a part of the blocking object; motion detection - to identify objects that move with respect to the background; texture and geometric shape - distinguishing a player, ball or shadow from a sign, slogan or graphic image etc . , and colour - and shades of colour.

The electronic exchange is preferably instant and unnoticeable by the viewer. Alternatively, it is possible to continuously "fade" the original image while enhancing the virtual image.

False identification of targets and images is unacceptable and must be avoided.

The algorithms preferably require special parallel processing hardware which will allow carrying out simultaneously the large number of operations involved in this process.

The method of this invention preferably uses two optional sub-systems: a) Digital Image Converter and Storage Unit -consisting of an electro-optical scanner for digital conversion and storage of virtual images, for constructing a memory unit for images such as advertisements. The system may also have the possibility of inputting images such as advertisements in other ways, as by digital interface (magnetic, optical disc, communication link) or video port, and may further include a graphics programme and man-machine interface for designing virtual images (like slogans) on-line. b) Locale "learning" and storage system - for creating a data bank of targets and fixed objects in locales such as stadiums and fields, including: signs (location, shape, colour and type - one-time, seasonal, etc.), court markers (lines, colour, goal/basket, post), etc .

These two sub-systems can operate off-line or can be part of the basic system. The system can "learn" the details of the court in the course of a live event and create/update its data bank for future use. This can also be done using the trial shots taken before the event starts .

The method involves the following steps: When the live or previously recorded video film is being transmitted, the following steps take place: 1 ) Frame grabbing and digitization - each video frame is grabbed and each pixel value is digitized and stored in system memory; 2) Searching - the captured video frame is scanned to detect either actual physical displays (like the icons stored in the memory) or background regions suitable for implantation whose specifications have been pre-defined. After detection, suspected targets, i.e. displays, are checked for unequivocal identification. This is accomplished by identification of messages and graphics in the displays, or of colour and texture attributes using standard pattern recognition techniques like edge detection, vertex detection, matching methods, character recognition, neutral network techniques and so on. After the target (display) has been identified, its optical magnification and orientation are calculated and the locations of all other stored targets (displays) in the frame are consecutively predicted using the locale's lay-out in the data bank, giving the system positive search clues for additional targets in the same video frame . 3) Blocked surface identification - when a given message area or display region is positively identified in a frame, the target (display) is compared with its properly scaled stored image (icon) and those areas of the display that are temporarily blocked by an object such as by the body of a player, by a ball or a shadow etc. are revealed. The pixel addresses of these surfaces are stored so that these surfaces will later be superimposed on the substituted image. 4) Scaling and substitution - when a physical image display or a free location is identified, the memory of the system is searched to find the desired virtual image to be substituted or implanted. The exchanged virtual image (patch) is then first normalized to acquire the proper size and perspective of the original physical image and identified blocked surfaces are then removed, so that the exchanged image looks like a background display or as a painted display on the court. 5) Real-time video tracking - typically a given display is visible for a few second before it moves out of the camera's field of view. The system preferably uses previous frames' information to track a given display throughout this succession of frames. To do that, conventional video tracking techniques, such as edge, centroid or correlation tracking methods, are executed. Tracking of players or of the ball can also be instrumental to identify blocking portions in the case where target icons are not stored in the system memory or for implantation in free regions.

BRIEF DESCRIPTION OF THE DRAWINGS Following is a brief description of the drawings which will help in understanding the invention.

Figure 1 is a logical flow diagram of the processes and tasks required in accordance with a preferred embodiment of the method of the invention; Figure 2 is a block diagram of the basic and sub-system modules in accordance with a preferred embodiment of the invention; Figure 3 shows a block diagram of a basic processing unit; Figure 4 illustrates a minimum basic on-line system in accordance with the invention; Figure 5 shows a minimum basic off-line system in accordance with the invention; and Figure 6 shows a system in accordance with the invention adapted for cable TV application.

Referring now to Figure 1 , this illustrates the process of this invention. In a preferred embodiment, the system and method are designed to perform the substitution of physical targets with synthetic images in real time, although a simpler version of the invention can be used off-line .

When operating the system, the modules required are illustrated in the block diagram of Figure 2. These include : a basic processing unit; an optional scanner /digitizer used to create the data bank of synthetic images from still pictures; and an optional sub-system composed of a TV camera, digitizer and memory to create the stadium data bank.

As was mentioned before, there may be other methods to create the data bank of synthetic images. The locale's (stadium's) data bank may also be created from the trial shots taken before the game starts or even be incrementally built in the course of the game by means of a "learning" algorithm.

Figure 2 illustrates a block diagram of the apparatus used in the system, wherein 1, 2, ....n are a plurality of TV cameras in different positions, 3 is the basic processing unit described in Figure 3, sub-system 4 converts and stores synthetic images and sub-system 5 is a "learning" and storage system for event locales and targets therein. The output 6 can be transmitted by cable, optical fiber or wirelessly. It can also be displayed and/or recorded.

The basic processing unit required to operate the system in real-time is shown in Figure 3. This module comprises : a frame grabber for colour image acquisition; a plurality of image memories; a fast parallel processor; a program memory; data banks of synthetic images to be substituted and of locale's lay-outs and target icons; a man/machine interface for control and for local display and recording; and an image digital to analog converter.

The above apparatus is used to locate in real time in each video frame, suitable areas within a stadium which have physical displays or might be suitable for embodying such displays, and to substitute for such physical displays, or introduce into such areas, virtual images which are stored in the memory of the system to serve as advertisements in the background. These electronic inserted images will be seen by viewers as if they are physical displays located in a stadium and all action taking place in front of the actual physical display will appear to the viewer to be taking place in front of the virtual image as well.

Figure 4 illustrates an on-line system in accordance with an aspect of this invention consisting of a video camera 10, video processing unit 12 and work station 14 that provides the required man/machine interface.

Figure 5 illustrates a basic off-line system in accordance with one aspect of this invention. In this case, a video tape 20 or a video cassette recorder is the input rather than a TV camera and this is processed by the processing unit 22 and work station 24 to provide a video tape output 26 with substituted images.

Figure 6 illustrates yet another application of the system of this invention, namely a cable TV center. The center 30 receives transmissions from stations 32 and 34. These transmissions are processed by the processing unit 22 and work station 24 and broadcast with substituted advertisements to subscribers from the center 30.

Although a preferred system according to this invention superimposes blocking objects and shadows on the virtual images, a less sophisticated and much cheaper system is also intended as part of this invention, and that is a system where virtual images are exchanged for physical without relating to blocking objects. Such a system can be quite useful for substituting images in unblocked regions, for example high up in a stadium.

From the above description of the invention, it is apparent that the system, method and apparatus described above can have many applications. Thus, it is also possible to introduce virtual images, such as slogans or graphic advertisement, on the uniforms of players, particularly when a player is shown in close-up. In such a case, the outline of the player, or at least his shirt or helmet, would be the target for implanting a virtual image .

Another possible application is the automatic generation of continuous video films showing only sequences wherein specific targets, which have been pre-selected, appear to the exclusion of sequences where these targets do not appear. Such video films can be useful for analyzing and monitoring the activity of specific targets, for example individual players and their performance throughout an entire team game. This enables tracking each individual throughout an entire game without having to replay the entire cassette for each player.

Another application of this invention is to generate statistical data of targets such as advertisements, for example the number of times an advertisement appears on the screen, and to debit acccordingly .

The implanted image can be in the form of a fixed, blinking or scrolling image, or it may be an animated film or video clip.

It will be appreciated by those skilled in the art that the invention is not limited to what has been shown and described hereinabove by way of example. Rather, the scope of the invention is defined solely by the claims which follow. 104725/2

Claims (2)

1. An electronic substitution system comprising : an electronic data bank of event locales and targets therein; means for detecting targets in digitized video frames; means for comparing at least one detected target with at least one desired target in the data bank; and means for superimposing a virtual image on a detected target which has been found by said means for comparing to be similar to said at least one desired target, and including means for replacing at least one target in a digitized video frame with at least one virtual image; and means for generating statistical data regarding operation of the means for replacing, and wherein said means for generating comprises means for computing the number of frames in which a particular target was replaced with a particular virtual image.

2. A system according to claim 1 and also comprising means for computing a monetary sum based on the statistical data. For the Applicant, C: 18467