WO2010049578A1

WO2010049578A1 - Video processing

Info

Publication number: WO2010049578A1
Application number: PCT/FI2009/000093
Authority: WO
Inventors: Mika Daniel Letonsaari
Original assignee: Mika Daniel Letonsaari
Priority date: 2008-10-28
Filing date: 2009-10-27
Publication date: 2010-05-06
Also published as: FI20090040A0

Abstract

A marking surface on an object is temporally optically alternated at one or more particular frequencies according to a given scheme. Video footage is captured of a scene with some parts of the marking surface appearing in the video footage. Also image detection frames are being captured of the scene at a rate different than in producing the video footage and/or in a frequency different than that in producing in the video footage. The marking surface is detected from the scene by detecting the temporal optical alternating in the image detection frames. Moreover, different marking surfaces can be identified from one another based on different temporal optical alternation schemes being detected from the image detection frames. A corresponding apparatus, system and computer program are also disclosed.

Description

VIDEO PROCESSING

FIELD OF THE INVENTION

The present invention generally relates to video processing. The invention relates particularly, though not exclusively, to marking an object in a video camera footage and modifying the video camera footage.

BACKGROUND

It is sometimes necessary to process a television image such that a part of the picture is replaced with supplementary content. For instance, weather forecasts are often produced by capturing a video of a weatherman explaining weather with a map that is not actually visible to the weatherman himself. Instead of a map, there is a chroma keyed layer behind the weatherman and the image is processed by replacing the chroma keyed area with desired content. However, this technique restricts the use of colors in the scene and its use is therefore mostly limited to studio environment.

Another application of video processing is adapting advertisements in international sports events. For instance, national laws may ban broadcasts with cigarette advertisements or advertisers wish to tailor their advertisements to different markets e.g. by using different texts and illustrations depending on target culture, language and business opportunities. It is known that a fixed advertisement may be specially illuminated such that it can be discerned by means of a camera that operates with infrared light. A unit of two cameras is then used to record the event such that both cameras have identical view, with one camera capturing the visible picture and another camera capturing a marking video using infrared light range in order to enable detecting of advertisements and their positions in the visible picture. The advertisements are then detected from the marking video and replaced with customized advertisements to the extent desired. However, in order to reliably detect the advertisements in the marking video, the illumination of the advertisements and their surroundings should be controlled such that there would be no excess unwanted infrared light or reflections of infrared light. While this may be possible indoors or generally in the absence of sunshine, there are often strong IR-rich light beams which may interfere with the detection of the advertisements. If the detection of an advertisement fails, it may be that no advertisement location is found or even worse, and more likely, the advertisement area may leak out to cover the picture of the actual event. While the advertisers may be unhappy if their paid advertisements fail to appear in a televised sports event, the audience may be even more disappointed if the opponent of a boxing match is covered by an advertisement that should only loom behind the boxer. Basically, any objects reflecting sufficient amount of infrared light, such as the skin, eyeballs, teeth, or clothes of a person might confuse the advertisement detection, if there is an excess of infrared light in the range captured by the marking camera.

It is also possible to manually edit captured videos off-line to mark advertisements and adapt their contents, but such a method is unfortunately not applicable to live broadcasts and in any case is very labor intensive and error prone work.

It is an object of the invention to mitigate problems related to prior art and/or to provide new alternatives.

SUMMARY OF THE INVENTION

According to a first aspect of the invention, a method is provided for marking an object in a video camera image, comprising temporally optically alternating a marking surface of an object according to an alternating scheme, capturing successive images of a scene covering the marking surface, detecting temporal optical alternation in the successive images and detecting the marking surface from the detected temporal optical alternation in the successive images.

The surface may be temporally optically alternated such that the surface appears to have a constant light intensity when seen by a human eye and by a camera that uses visible light, whilst the image detector can identify the surface h baasseerdi n onn t thhee t tiimmee v vaarrvyiinnng i illlluummiinnaattiioonn.

The marking of the object may comprise illuminating the object with periodical invisible or visible light at instants synchronized with image capture periods of the video image footage.

A high speed television camera may be used to capture the video camera footage and the successive images of the scene. In this case, the frame rate of the video camera footage may be reduced from a higher original rate. In an alternative embodiment the successive images of the scene and the video camera footage may be captured using separate objectives. In yet another alternative embodiment the successive images of the scene and the video camera footage may be captured using a common objective and a beam splitter.

The marking surface may be temporally optically alternated by means of a particular sequence. The sequence may be used to identify different marking surfaces. Moreover, different marking surfaces may be distinguished by using combinations of different independent temporal optical alterations. The different independent temporal optical alterations may involve changes in polarity, frequency, or frequency distribution.

The marking surface may comprise a particular visual pattern. The pattern may comprise inner edges starting from rectangle corners. Where only part of the marking surface is visible, any visible inner edges together with the outer edges of the rectangle may be extrapolated and corner places may be estimated to be on edge intersections.

The method of the first aspect may provide significant advantages over prior known technologies. For instance, in chroma keying technique marking surfaces are colored with a predetermined color. The marking color cannot be used elsewhere in the scene in order to enable recognizing an object (such as the marking surface) by their color. This technique restricts the use of colors in the scene and its use is therefore mostly limited to studio environment. Using static invisible light to distinguish marking surfaces enables normal appearance of all visible colors in the scene. However, even when invisible light or invisible components of light are used, controlled lighting conditions are still required. For example, sunlight or reflections of sunlight may easily pose problems, because sunlight contains a strong infrared component. On the contrary, using temporal optical alternating of the marking surface may not similarly restrict the lighting conditions, because there is no high frequency oscillating light in nature. In terms of real-life illumination, significantly enhanced object recognition reliability may be achieved. Moreover, the use of different alternating sequences may further enable simultaneous identifying of different marking surfaces. Using sequences to construct certain visual patterns on marking surfaces may further enhance the object recognition reliability.

According to a second aspect of the invention there is provided an apparatus for detecting a marking surface in video camera footage by using successively captured images of a scene covering the marking surface, the apparatus comprising: a memory comprising instructions; and a processor configured to operate according to the instructions and accordingly to: detect temporal optical alternation in the successive images; and detect visible parts of the marking surface in the video camera footage from the identified temporal optical alternation in the successive images corresponding to a temporal optical alternating scheme.

The scheme may be temporally repeated. The apparatus may be further configured to cause the temporal optical alternating of the marking surface.

The detecting of the visible parts of the marking surface from the identified temporal optical alternation corresponding to a temporal optical alternating scheme may be based on detecting the presence of the temporal optical alternation from a particular series of captured images of the scene.

The marking surface may be provided on an object. The object may be visible or invisible to human eye. An invisible object may be made of transparent material.

The marking surface may be temporally optically alternated such that the marking surface appears to have a substantially constant light intensity when seen by a human eye and by a camera that uses visible light, whilst the image detector can detect the surface based on the temporal optical alternation of the marking surface that may result, for instance, from illumination by alternating light.

The temporal optical alternating of the marking surface may comprise illuminating the object with periodical invisible or visible light synchronized with video camera image capturing.

A high-speed television camera may be used to record the video camera footage and also to provide the successively captured images of the scene. In this case, the frame rate of the production video may be reduced from the frame rate at which the high-speed television camera operates. In an alternative embodiment the successively captured images of the scene may be captured using a separate objective or using a common objective with capturing of the video camera footage and a beam splitter. The marking surface may be temporally optically alternated by means of a particular sequence. The sequence may be used to identify different marking surfaces. Moreover, different marking surfaces may be distinguished by using combinations of different independent temporal optical alterations. The different independent temporal optical alterations may involve changing of polarity, frequency, or frequency distribution.

According to a third aspect of the invention there is provided a method for detecting a marking surface in video camera footage by using successively captured images of a scene covering the marking surface, comprising: detecting temporal optical alternation in the successive images; and detecting visible parts of the marking surface in the video camera footage from the identified temporal optical alternation in the successive images corresponding to a temporal optical alternating scheme.

The method of the third aspect may further comprise temporal optical alternating of the marking surface.

The marking surface may be temporally optically alternated such that the marking surface appears to have a substantially constant light intensity when seen by a human eye and by a camera that uses visible light.

The detecting of the visible parts of the marking surface from the identified temporal optical alternation corresponding to a temporal optical alternating scheme may be based on detecting the presence of the temporal optical alternation from a particular series of captured images of the scene. The temporal optical alternating of the marking surface may comprise illuminating the object with periodical invisible or visible light synchronized with video camera image capturing.

The marking surface may be temporally optically alternated using a particular sequence. The sequence may be used to identify different marking surfaces. The different marking surfaces may be distinguished by using combinations of different independent temporal optical alterations.

The different independent temporal optical alterations may comprise changes of polarity, frequency, or frequency distribution.

The marking surface may comprise a particular visual pattern.

The pattern may comprise inner edges starting from rectangle corners.

According to a fourth aspect of the invention there is provided a system comprising a controller according to the second aspect and any one of the following: a marking surface alternator configured to temporally optically alternate the marking surface; a first camera configured to capture the video camera footage; a second camera configured to successively capture images of a scene covering the marking surface; and the marking surface.

The marking surface alternator may comprise controllable illumination equipment.

According to a fifth aspect of the invention there is provided a computer program stored on a computer readable memory medium for controlling an apparatus detecting a marking surface in video camera footage by using successively captured images of a scene covering the marking surface, the computer program comprising: computer executable program code configured to enable the apparatus, when executed by the apparatus, to detect temporal optical alternation in the successive images; and computer executable program code configured to enable the apparatus, when executed by the apparatus, to detect the marking surface from the identified temporal optical alternation in the successive images corresponding to the temporal optical alternating of the marking surface.

The computer program stored on a computer readable memory medium according to the fifth aspect may further comprise computer executable program code configured to enable the apparatus, when executed by the apparatus, to perform any method according to any embodiment of the third aspect.

It is appreciated that any embodiment of a particular aspect or as described in the detailed description may also be applied to another aspect even though not all such combinations would be expressly presented in this summary.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following, the invention will be described in detail with reference to the attached drawings, wherein

FIG. 1 presents a schematic diagram representing a system according to an embodiment of the invention;

FIG. 2 presents a schematic diagram representing a system according to another embodiment of the invention;

FIG. 3 presents a block diagram representing an imaging system according to an embodiment of the invention; FIG. 4 presents an exemplary image taken of a particular scene with a high speed video camera, when a marking surface is not illuminated;

FIG. 5 presents an exemplary image taken of a particular scene with the high speed video camera, when the marking surface is illuminated;

FIG. 6 presents a difference image based on the difference of images FIG. 4 and FIG. 5;

FIG. 7 presents a sample image to be inserted in video stream;

FIG. 8 presents a modified sample image of Fig. 7 scaled, positioned and masked according to the difference image of Fig. 6;

FIG. 9 presents a modified video image, where the modified sample image of FIG. 8 is inserted into original video image FIG 5;

FIG. 10 presents a guiding pattern for marking an object according to an embodiment of the invention; and

Fig. 11 presents a block diagram of a unit for detecting a marking surface from signals received from a camera unit.

DETAILED DESCRIPTION OF THE INVENTION

Fig. 1 presents a schematic diagram representing a basic system 100 according to an embodiment of the invention. The basic system comprises an object with a marking surface 1 , a lamp 2 adapted for illuminating the marking surface, a camera unit 3 for capturing video image of a scene that here comprises the marking surface and for detecting the marking surface, and an obstacle 4 that partly masks or visually blocks the marking surface 1 from the camera unit 3. The camera unit 3 comprises as different functional blocks a video image camera 5 and an image detection camera 6. The video image camera 5 and the image detection camera 6 may comprise two separate optical light capture elements or a common light capture element. The basic system 100 further comprises a controller 1000 for controlling the operation of the camera unit and possibly for synchronizing the operation of the camera unit 3 and the lamp 2 by steering the timing of either or both of the camera unit 3 and of the lamp 2. Different structures and operations will be explained in further detail in the following.

The controller 1000 may generally control the operation of the camera unit 3 and of the lamp 2. In this description, these operations are generally described as they occur without separate references to the source of corresponding instructions from the controller 1000. This simplification is intended to make this description easier to read. However, it should be appreciated that there may be a separated or integrated controller 1000 that may consist of one or more parts (e.g. distributed to different camera units and/or lamps 2) and that actually decides upon various operations.

Fig. 1 facilitates the understanding of particular embodiments of the invention. Fig. 1 does not, however, limit the scope of the invention. To the contrary, various elements of Fig. 1 may be implemented differently. For instance, the lamp 2 is used to periodically alter the optical appearance of the marking surface. However, the optical appearance of the marking surface may alternatively be manipulated to change its reflectance, transparency, or emission. Each of the reflectance, transparency or emission may be altered in terms of intensity, frequency range, polarization angle, or any combination thereof. For instance, Liquid Chrystal Displays (LCD) employ electric altering of polarization of liquid crystals in an electric field such that the reflectance may rapidly change. Also Micro-Electro-Mechanical Systems (MEMS) can be used to periodically move reflective or refractive optical elements (e.g. mirrors, prisms, and lenses) such that the appearance of the marking surface periodically alternates in a manner which enables automatic detection. It is appreciated that e.g. using polarization changes, the image detection camera 6 is correspondingly adapted to record frames which enable detecting corresponding differences. In this example of polarization changes, the image detection camera 6 may have a polarization filter. These alternative ways to reflect the appearance of the marking surface are no less useful than changing the illumination with the lamp 2, but it is simplest to explain different embodiments with reference to using a lamp and thus the following description focuses to such an implementation without intention to exclude any other techniques. Moreover, the lamp may be built in an object that carries the marking surface. For instance, the lamp 2 may comprise a plurality of Light Emitting Diodes (LEDs) and the marking surface may comprise a transparent or translucent cover such as a textile. The cover may present desired content such as one or more advertisements.

Let us consider further the embodiment presented in Fig. 1. The marking surface 1 may comprise a fixed advertisement of rectangular form. The video image camera 5 may be a video camera or television camera such as, for example, a National Television System Committee (NTSC) standard compliant camera. The video image camera 5 is generally capable of producing video footage at a production video frame rate. The production video frame rate may be, for instance, 30 frames per second (fps).

In a first embodiment, the video image camera 5 is a high-speed video camera configured to capture video images at a rate of 60 fps, twice the frame rate of desired production video. Basically, in this embodiment of the invention, a video image camera 5 operates additionally as an image detection camera 6 that takes at least two measurements during each production video frame interval. A marking surface is identified on the basis of different light intensity between the measurements, all other objects remaining at about constant light intensity. The areas of the object visible in the image frames are then determined by combining this information with a known relationship between the picture coordinates of video image camera 5 and those of the image detection camera 6. The image corresponding to the visible parts of the object may then be modified in a predetermined manner.

In an embodiment, the image detection camera 6 is configured to capture video images at a frame rate higher than the production video frame rate, say at four or six times the frame rate of the production video frame rate. The heightened frame rate of the image detection camera 6 may be utilized for various needs. First, a higher frame rate can be used to perform more image detections per one production video image frame so that any marking surfaces can be more reliably detected and/or motion in the video image frames may be better compensated when there is more information to compensate detected motion. Such compensation information is usable for compensating any shift in the position, orientation or scale of a marking surface 1 or of any obstacles 4 masking the marking surface 1. Second, the higher frame rate can be used to distinguish different marking surfaces based on different optical appearance alternating patterns. Let us assume that four image detection frames are captured per one video image frame. Then, it is possible to distinguish between six different appearances of the marking surface 1 with equal off and on times during one image interval. By using binary code representation, the different patterns would be 0011 , 0101 , 0110, 1001 , 1010 and 1100. Equal off and on times result in better intensity contrast than unequal times. However, if more alternatives are needed, it suffices that any one of the four image detection frames differs from other image detection frames, i.e. all combinations except 0000 and 1111 may be used. This results in 16-2 = 14 different alternatives with frame rate ration of 4. Correspondingly, with frame rate ratios of 5, 6, 8 and 10 the respective number of different alternatives would be 30, 62, 254 and 1022. In other words, a large number of different marking surfaces may be relatively simply distinguished without necessarily modeling the locations of the marking surfaces and the current scene of the camera unit 3. It is appreciated that it is possible to similarly provide different patterns for alternating the appearance of the marking surface in other embodiments of the invention and in particular in the second and third embodiments.

FIG. 2 presents a schematic diagram representing a system 200 according to a second embodiment of the invention, in which the camera unit 3 comprises as separate and parallel light capture elements a video image camera 5 and an image detection camera 6. The video image camera 5 is configured to capture production video footage for optional further processing. The image detection camera 6 is configured to capture image detection frames for identifying objects that are illuminated with light that pulsates at a predetermined rate on the frequency area in which the image detection camera 6 operates.

The system 200 of Fig. 2 is subjected to a parallax offset that may be compensated. The compensation may employ, for instance, computational estimating and compensating of the location of the marking surface detected by the image detection camera 6 before or on adapting the captured video footage.

In a particular variant of the second embodiment, the image detection camera 6 is configured to capture images at the production video frame rate and using visible light, but with a time offset from the video image camera 5. The time offset may range between the exposure time of the video image camera (e.g. 1/200 s) and a half cycle of the video image camera. This variant may be constructed using two normal and similar video image cameras so that common spare parts can be used for each camera and sourcing may be simplified. Moreover, in this embodiment, the timing of the video image camera 5 and of the image detection camera 6 can brought very close to each other so that pairs of video image frames can be captured within instantaneous time and fast moving objects cause less errors to the detection of the marking surfaces than if the time difference is larger. This variant may require, however, particular synchronization between the camera unit 3 and the pulsating illumination of the marking surface in order to change the appearance of the marking surface 1 between the image capture by the video image camera 5 and by the image detection camera 6.

FIG. 3 presents a block diagram representing an imaging system 300 according to a third embodiment of the invention. In this system the camera unit 3 has a common objective 31 for both a video image camera element 36 and an image detection camera element 37. In Fig. 3, light arriving from the scene passes through the objective 31 and travels as a light beam 32 to a beam splitter 33. The beam splitter 33 divides the light beam 32 into two beams, first sub-beam 34 and a second sub-beam 35. The first sub-beam comprises visible light from the scene and is directed to the video image camera element 36 that functionally corresponds to the video image camera 5. The second sub-beam 35 carries light to the image detection camera element 37. The beam splitter may be wave length selective i.e. the first and second beams may have light of different frequency ranges from a common scene. In this case, the second sub- beam 35 may comprise light that is invisible to human eye.

The image detection camera element 37 may be configured to operate at a frame rate that is a multiple of the frame rate of the video image camera element 36. Alternatively, if the image detection camera element 37 is configured to operate in a common frequency area with the video image camera element 36, the image detection camera element 37 may be configured to operate at a common frame rate with the video image camera element 36 in a manner similar to that explained for the variant of the second embodiment.

The operation of different embodiments of the invention is next further described by reference to Figs. 4 to 10.

Referring to Fig. 1 , the lamp 2 is configured to radiate e.g. 30 Hz pulsed infrared light behind the advertisement so that the radiation permeates the entire area of the advertisement. The camera unit 3 and the lamp 2 may be synchronized so that the lamp is off and on when alternating video images are captured by the video image camera 5 as subsequent video image frames. The video image frames captured with the lamp 2 switched off are here denoted as odd frames and illustrated in Fig. 4 while the video image frames captured with the lamp 2 switched on are denoted as even frames and illustrated in Fig. 5. When the lamp 2 is switched off, the marking surface appears in high-speed detection (whether in visible or invisible frequency area) alike the obstacle 4. On the other hand, when the lamp 2 is on, the marking surface 1 appears to the image detection camera 6 as illuminated. It should be appreciated that other details of the scene are suppressed in Figs. 4 to 9 in order to simplify this description. However, it will also become clear to a skilled reader that any further details in the region surrounding the marking surface are simply not affected by a process of embodiments described in this document and that such details in the scene have little or no impact to the operation of the described embodiments.

Figs. 4 and 5 present consecutive images as seen by the image detection camera (in a simplified manner). The term image detection camera refers to any equipment for this purpose. For instance, the image detection camera may be a high-speed video image camera 5 shown in Fig. 1 as well as a separate image detection camera shown in Fig. 2, an image detection camera element 37 shown in Fig. 3. Let us assume here that the marking surface is identified by means of a pulsating visible light. In this case, ambient and constantly visible light is reflected by the obstacle 4 which thus appears in high-speed image detection as an unchanged shape 44 in Figs. 4 and 5. It is understood that any other details in the image whether in front of or around the marking surface 1 would appear unchanged by the alternating appearance of the marking surface 1. On the other hand, the marking surface 1 appears in Fig. 4 rather similarly to the obstacle 4 as a dark rectangle 41 that is merged with the shape 41 (of the obstacle 4) when the lamp 2 is off. Notice that in this document dark areas are represented by dotted areas. When the lamp 2 is on, the marking surface appears as a light radiating source and is represented as a light rectangle 51. It is appreciated that the processing of video images may be based on common video processing techniques. For instance, a multichannel color image can be converted into a single channel intensity image if needed. There are also well known techniques for converting a multichannel color image into a single channel intensity image, hence these techniques are not described in detail here. The technique for conversion may be chosen such that it maximizes the marking surface contrast in consecutive images.

Next, an absolute intensity difference of two consecutive images is computed. In the example presented in Fig. 6, those areas where illumination has changed have a high value (light) and other areas have low value (dark). Because there is no high frequency oscillating light in nature, a new image 61 results that practically corresponds to visible parts of intentionally created marking surfaces. The new image 61 clearly illustrates a rectangular area 62 that corresponds to the visible parts of the marking surface 1.

An image of alternative content 71 such as an advertisement image shown in FIG. 7 is stored in the controller 1000 or in a separate computer. A rectangular form of the marking surface image 62 is identified in the new image 61 of FIG. 6 and the location of the alternative content is determined by using pattern recognition algorithms. This determination may employ particular patters in the new image 61 as exemplified by Fig. 10 and further described with reference thereto. The alternative content is scaled and positioned according to the identified rectangle. Using the difference image presented in FIG. 6, the scaled and positioned alternative content is masked so that only visible parts of the marking surface 1 contain respective parts of the scaled and positioned alternative content. Resulting image is presented in FIG. 8. The image of Fig. 8 is then added into corresponding original odd image shown in FIG. 4 in which the marking surface 1 is not highlighted. The adding results in a combination image presented in FIG. 9, wherein visible parts of the marking surface are replaced with corresponding parts of suitably scaled and positioned alternative content. This final image is used in the production video. As can be seen, in this example every consecutive image pair is processed to produce one image thus reducing 60 Hz frame rate down to 30 Hz. If a separate image detector is used for production video, for example if images in FIG. 4 and FIG. 5 are obtained from infrared image detector, the image presented in FIG. 8 is added to the production video image which temporally corresponds to the image presented in FIG. 4, resulting in a image presented in FIG. 9.

It is appreciated that the blinking of the lamp 2 at a high enough rate makes the light of the lamp appear to a human eye as a consistent illumination for the marking surface 1.

It is also appreciated that in principle, the lamp 2 only has to illuminate the marking surface during every second period in which image detection is carried out. In fact, the illumination may only cover a part of the duration of the illuminated image detections, if all or substantial portion of the image is captured simultaneously such that reduced illumination time will not distort the form of the detected image. In this case, the effect of the illumination is reduced but may yet suffice. The illumination may be thus reduced under such an ambient light in which a reduced illumination suffices to reliably identify the marking surface. As an advantage, the glaring caused by the advertising surface and / or energy consumption and heating effects may be reduced. However, the timing of the illumination by the lamp 2 has to be synchronized with the capturing periods of the high-speed detection.

In an alternative embodiment, the lamp 2 is configured to illuminate the marking surface periodically such that the lamp is on and off alternating cycles at the image detection frame rate (e.g. on 1/60 seconds and then off 1/60 seconds) in a recurring manner. In this case, the lamp 2 illuminates subsequent frames in differing manner unless the illumination cycle starts after one half of the highspeed image capturing period has lapsed. Hence, sufficient synchronization may be achieved e.g. by recording video image from a scene with a marking surface and automatically detecting whether the rectangle of Fig. 6 is identifiable. If not, the recording is time-shifted by a suitable time e.g. by one half of the exposure period of the image detection. Let us assume that initially the lamp 2 and the image detection are most inconveniently synchronized to one another, i.e. the illumination of the lamp 2 lasts one image detection cycle and starts exactly in the middle of a image capture period of one video image frame (and thus ends exactly in the middle of the following image detection capture period). In this scenario, the rectangle visible in Fig. 6 will not appear. Next, the timing of the video image camera and image detection camera is shifted by one half of the image detection cycle, say by 1/200 s i.e. by 5 ms. Now, the illumination by lamp 2 starts simultaneously with image capture period of one video detection frame and stops simultaneously with the end of that image detection frame or with the start of the subsequent image detection frame that is then the frame without illumination. Then, the differential image shown in Fig. 6 has a clearly identifiable rectangle. Moreover, it is easy to detect which one of the two image detection frames is the one without illumination by the lamp 2 in order to suitably determine which of the image detection frames shall be used as basis for combining the alternative content 71.

Referring to the example described in the preceding paragraph, let us further assume another scenario in which the lamp 2 and the camera are initially temporally aligned such that the illumination period of the lamp 2 starts one quarter after the start of an image detection frame, and the illumination period lasts for one image detection cycle (e.g. 1/60 s). In this case, the rectangle in Fig. 6 is identifiable, but not necessarily sufficiently well. The image detection and camera operation may then be delayed by one half of the detection image capture period. In result, the illuminated part will still cover one quarter of the image capture period in one frame and three quarters in adjacent image detection frames. Hence, the rectangle is not clearer to identify. Next, a second similar delay is caused to the operation of the video image camera and to the image detection camera. This second delay shifts the image detection so that the illumination of the marking surface starts one quarter of the detection period before one image detection frame starts. Thus, the illumination should completely cover the image capture period in one frame but not in its adjacent frames and the rectangle in Fig. 6 should be clearly identifiable. The timing of the high-speed image detection may also be automatically adjusted in real time. Additional high-speed image detection equipment may be provided to test different time offsets. In such an embodiment, it may be possible that different marking surfaces are illuminated with independent lamps 2 at random time offsets. It may even be possible to distinguish from a common scene different marking surfaces based on their different time offsets without necessarily synchronizing the operation of the different lamps, based on likely timing difference incurred by random time offsets. In this case, if two marking surfaces would, by co-incidence, become synchronized on start up or due to drifting during operation, the synchronization could easily be removed by switching on and off one or more of the lamps 2 so that new time shifts would be randomized.

In an alternative embodiment, there is provided a synchronization circuitry for synchronizing the lamp 2 or a plurality of lamps 2 with the camera and highspeed image detection.

In the preceding exemplary embodiments it was described how image detection may operate at a rate that is a multiple of the video image frame rate. It should be appreciated that a high frame rate in the image detection may be advantageous for accurate locating of visible parts of the marking surface 1 when obstructed by fast-moving obstacles. However, in an alternative embodiment, the image detection employs image detection frames captured at equal or lower rate than that of video image capture. If visible light is used in the image detection, the even and odd frames may refer to different successive video image frames. For instance, the appearance of the marking surface 1 may be alternated between each subsequent video image frame. In this case, the appearance would effectively alternate on frequency of 15 Hz and the marking surface 1 might appear as blinking in the production video. However, this effect may be computationally remedied by suitably compensating the luminance of the visible parts of the marking surface 1. Moreover, in case that these visible parts are replaced by the alternative content, the possible alternating of the luminance would also become addressed. Further alternatively, the image detection may use a frame rate such as 1/3 of the video image frame rate. In this case, some image detection frames coincide with video frames while some other image detection frames do not coincide with video frames. Hence, it is possible to maintain the appearance of the marking surface 1 stable in these coinciding frames so that the production video should not appear blinking to viewers. On the other hand, if an invisible frequency range is used for the image detection, the image detection may take place in any frequency. It may be advantageous to yet coincide some image detection frames with the image frames in order to enhance the accurate locating of the visible parts of the marking surface in each image frame. However, the coincidence is not necessary, but then the position of moving obstacles or visible areas of the marking surface 1 may be estimated by means of interpolation or extrapolation.

FIG. 10 presents an example of a recognition pattern 101 on the marking surface 1. The pattern 101 comprises two patterns drawn with and without dotting in FIG. 10. The two patterns may be altematingly illuminated in opposite phases or one of the patterns may be intermittently illuminated while the remainder is not. When one pattern is illuminated, the remaining one is not. This allows a good detection of not only the outer edges 11 , but also the inner edges 12 to 15. The inner edges are aligned so that they all lead to corner points of the rectangular object. If large parts of the object are occluded, the corners of the marking surface can be estimated by extrapolating known edge parts. If for example only quarter of the marking surface is visible, i.e. the part between points 16 and 17, extrapolating lines 14 and 15 together with the known outer edge between points 16 and 17 defines the location of the left side corner points. Right side corner points are also found based on symmetry of the pattern 101 so that the location of the whole marking surface becomes known. The angles of the inner edges are selected so that they divide the opposite outer edge into known fractions. For example, the inner edge 12 divides the bottom edge in parts ^λA and ³A, and the inner edges 13 and 15 divide opposite edges in half. Since perspective distortion appears as linear transformation, extrapolation of edges into corner points and known points in outer edges of rectangular object also stays valid.

Fig. 11 shows a controller unit 1000 for detecting a marking surface from signals received from a camera unit. The controller unit 1000 may be configured to operate as the controller 3. The controller unit 1000 may be based on personal computer architecture. For instance, the controller unit 1000 may be based on a laptop computer or on a desktop computer, or on a networked server such as an internet server. The controller unit 1000 comprises a processor 110 such as a central processing unit or a digital signal processor, and functionally connected to the processor a memory 120, an input/output element 150, a user interface 160, a power supply 170, and an optional synchronizing circuitry 180.

The memory 120 comprises a work memory 130 and a persistent memory 140 comprising an operating system 141 and software 142. The operating system 141 provides basic functionalities to the controller unit 1000 while the software 142 comprises particular instructions for controlling the processor 110, when executed by the processor, to different operations needed to implement particular embodiments of the invention. It is appreciated that the memory 120 need not be a singular unit. To the contrary, the persistent memory 140 and the work memory 130 need not be contained in a common module. Moreover, either or both of the persistent memory 140 and work memory 130 may comprise two or more different units which may also differ from one another by their types. It is also appreciated that the controller unit 1000 need not always contain the software 142, but may be delivered without the software and configured to receive the software from an external source such as a data network or transferrable memory medium. In general, the controller unit comprises the software 142 on some memory medium when operating according to an embodiment of the invention. The input/output element 150 comprises equipment for exchanging information with other elements such as the lamp 2 and the camera unit 3. The input/output element 150 may comprise, for instance, a network interface device, a digital / analog converter, a digital input bank, or any combination of thereof. The user interface 160 may comprise information presentation and/or input devices in order to enable a user to control and/or monitor the operation of the controller unit 1000. The power supply may comprise a battery, a mains transformer, a direct current transformer, a direct current connector (e.g. a Universal Serial Bus, USB, or a Firewire connector), or any combination thereof. The function of the power supply is to power the circuitries of the controller unit 1000.

The synchronizing circuitry 180 may be provided as a separate unit that may or may not be functionally connected to other parts of the controller 1000. The synchronizing unit may comprise a comparator (not shown) configured to compare timing of the lamp 2 and of the camera unit 3 and to responsively control the timing of either or both of the lamp 2 and of the camera 3.

The controller may further comprise a separate processor or be configured to use its processor 110 or other circuitry to modify the captured video footage in the visible areas of the marking surface. Such modification may involve overlaying or superimposing alternative content on the visible parts, highlighting the visible parts (e.g. by increasing luminosity.alternating luminosity and/or modifying different chrominance components for the visible parts), or any image manipulation known in the field of video processing.

What has been described and illustrated herein is a preferred embodiment of the invention along with some of its variations. The terms, descriptions and figures used herein are set forth by way of illustration only and are not meant as limitations. Those skilled in art will recognize that many variations are possible within the scope of the invention in which all terms are meant in their broadest reasonable sense unless otherwise stated. Any headings are used in this document solely for convenience of the readers with no intended limiting effect on the scope of claims which alone define the protection sought.

Claims

Claims:

1. A method for marking an object in a video camera footage, comprising optically alternating a marking surface of an object according to an alternating scheme, capturing successive images of a scene covering the marking surface, detecting optical alternation in the successive images and detecting the marking surface from the detected optical alternation in the successive images.

2. A method according to claim 1 , wherein the surface is optically alternated such that the surface appears substantially constant to a human eye and by a camera that uses visible light.

3. A method according to claim 1 or 2, wherein the marking of the object comprises illuminating the object with periodical invisible or visible light at instants synchronized with image capture periods of the video camera footage.

4. A method according to any one of the preceding claims, wherein a high speed television camera is used to capture the video camera footage and the successive images of the scene.

5. A method according to claim 4, wherein the frame rate of the video camera footage is reduced a higher original frame rate.

6. A method according to any one of claims 1 to 3, wherein the successive images of the scene and the video camera footage are captured using separate objectives.

7. A method according to any one of claims 1 to 3, wherein the successive images of the scene and the video camera footage are captured using a common objective and a beam splitter.

8. A method according to any one of the preceding claims, wherein the marking surface is temporally optically alternated by means of a particular sequence.

9. A method according to claim 8, wherein the sequence is used to identify different marking surfaces.

10. A method according to claim 8 or 9, wherein the different marking surfaces are distinguished by using combinations of different independent optical alterations.

11. A method according to claim 10, wherein the different independent optical alterations may involve changes in polarity, frequency, or frequency distribution.

12. A method according to any one of the preceding claims, wherein the marking surface comprises a particular visual pattern.

13. A method according to claim 12, wherein the pattern comprises inner edges starting from rectangle corners.

14.An apparatus for detecting a marking surface in video camera footage by using successively captured images of a scene covering the marking surface, the apparatus comprising: a memory comprising instructions; and a processor configured to operate according to the instructions and accordingly to: detect temporal optical alternation in the successive images; and detect visible parts of the marking surface in the video camera footage from the identified temporal optical alternation in the successive images corresponding to a temporal optical alternating scheme.

15.An apparatus according to claim 14, wherein the processor is further configured to cause the temporal optical alternating of the marking surface.

16.An apparatus according to claim 15, wherein the marking surface is temporally optically alternated such that the marking surface appears to have a substantially constant light intensity when seen by a human eye and by a camera that uses visible light.

17.An apparatus according to any one of claims 14 to 16, wherein the detecting of the visible parts of the marking surface from the identified temporal optical alternation corresponding to a temporal optical alternating scheme is based on detecting the presence of the temporal optical alternation from a particular series of captured images of the scene.

18.An apparatus according to claim 17, wherein the temporal optical alternating of the marking surface comprises illuminating the object with periodical invisible or visible light synchronized with video camera image capturing.

19.An apparatus according to any one of claims 14 to 18, wherein the marking surface is temporally optically alternated using a particular sequence.

20.An apparatus according to claim 19, wherein the sequence is used to identify different marking surfaces.

21.An apparatus according to claim 19 or 20, wherein the processor is configured to distinguish different marking surfaces by using combinations of different independent temporal optical alterations.

22.An apparatus according to claim 21 , wherein the different independent temporal optical alterations comprise changes of polarity, frequency, or frequency distribution.

23.An apparatus according to any one of claims 19 to 22, wherein the marking surface comprises a particular visual pattern.

24.An apparatus according to claim 23, wherein the pattern comprises inner edges starting from rectangle corners.

25.An apparatus according to any one of claims 14 to 24, further configured to modify in a predetermined manner the video footage for the areas corresponding to the visible areas of the marking surface.

26.An apparatus according to claim 25, wherein the modification comprises replacing alternative content in place of the visible areas of the marking surface.

27.A method for detecting a marking surface in video camera footage by using successively captured images of a scene covering the marking surface, comprising: detecting temporal optical alternation in the successive images; and detecting visible parts of the marking surface in the video camera footage from the identified temporal optical alternation in the successive images corresponding to a temporal optical alternating scheme.

28.A method according to claim 27, further comprising temporal optical alternating of the marking surface.

29.A method according to claim 28, wherein the marking surface is temporally optically alternated such that the marking surface appears to have a substantially constant light intensity when seen by a human eye and by a camera that uses visible light.

30. A method according to any one of claims 27 to 29, wherein the detecting of the visible parts of the marking surface from the identified temporal optical alternation corresponding to a temporal optical alternating scheme is based on detecting the presence of the temporal optical alternation from a particular series of captured images of the scene.

31. A method according to claim 30, wherein the temporal optical alternating of the marking surface comprises illuminating the object with periodical invisible or visible light synchronized with video camera image capturing.

32.A method according to any one of claims 27 to 31 , wherein the marking surface is temporally optically alternated using a particular sequence.

33.A method according to claim 32, wherein the sequence is used to identify different marking surfaces.

34.A method according to claim 32 or 33, further comprising distinguishing different marking surfaces by using combinations of different independent temporal optical alterations.

35.A method according to claim 34, wherein the different independent temporal optical alterations comprise changes of polarity, frequency, or frequency distribution.

36.A method according to any one of claims 32 to 35, wherein the marking surface comprises a particular visual pattern.

37.A method according to claim 36, wherein the pattern comprises inner edges starting from rectangle corners.

38.A method according to any one of claims 27 to 37, further comprising modifying in a predetermined manner the video footage for the areas corresponding to the visible areas of the marking surface.

39.A method according to claim 38, wherein the modification comprises replacing alternative content in place of the visible areas of the marking surface.

40. A system comprising an apparatus according to the second aspect and further comprising any one of the following: a marking surface alternator configured to temporally optically alternate the marking surface; a first camera configured to capture the video camera footage; a second camera configured to successively capture images of a scene covering the marking surface; and the marking surface.

41. A system according to claim 40, wherein the marking surface alternator comprises controllable illumination equipment.

42.A system according to claim 40 or 41 , further comprising a processor configured to modify in a predetermined manner the video footage for the areas corresponding to the visible areas of the marking surface.

43.A system according to claim 42, wherein the modification comprises replacing alternative content in place of the visible areas of the marking surface.

44.A computer program stored on a computer readable memory medium for controlling an apparatus detecting a marking surface in video camera footage by using successively captured images of a scene covering the marking surface, the computer program comprising: computer executable program code configured to enable the apparatus, when executed by the apparatus, to detect temporal optical alternation in the successive images; and computer executable program code configured to enable the apparatus, when executed by the apparatus, to detect the marking surface from the identified temporal optical alternation in the successive images corresponding to the temporal optical alternating of the marking surface.

45.A computer program according to claim 44 stored on a computer readable memory medium, further comprising computer executable program code configured to enable the apparatus, when exe.cuted by the apparatus, to perform a method according to any one of claims 27 to 39.