DE19925159A1 - Generation of panoramic videos using modular panoramic video camera, involves forming sum of individual videos produced by several video camera heads, and correcting e.g. axial steps - Google Patents

Abstract

The method involves producing the sum of a number of individual videos produced by an arrangement with several video camera heads (b) and correcting for the resulting axial steps, distortions and colour differences to combine the images together without visible transitions An Independent claim is also included for an arrangement for generation of panoramic videos with several camera heads.

Description

field of use

The invention is an apparatus and method for generating digital panoramic video (image, sound) according to the Preamble of claim 1 + 2.

State of the art

All-round videos / films have been a challenge since the introduction of moving images and have been conditional for two decades State of the art.

These existing all-round films are shot with special cameras or special wide-angle lenses, the film is on finally made visible again with a special projector.

The IMAX process as the best known (and qualitatively best) requires an approx. DM 1 million 70 mm film camera special Nikon 220 degree wide-angle lens (which unfortunately is no longer manufactured) and handmade 70 mm film material al. Subsequent digital processing is only possible with extremely high financial expenditure. There are only two Fir worldwide men who can digitize such material. Prices are only available after consultation about the entire order in any case over 40 DM per frame, d. H. about 1000 DM per second of film. The resulting image size of 48 megabytes (that means 1.125 gigabytes per second) also places very high demands on the system with which the digitized Ma material to be processed.

Another method uses an Altavista camera instead of the 70 mm camera whose film material and format are the same of the slide film resembles. The Nikon lens mentioned above is also used here. The digitization costs are here about 50% of the above Amount. Image size and demands on the finishing or processing system remain the same.

A disadvantage of both of the above Systems is the short recording time (approx. 3 min.) And the additional effort of sound in this surround to integrate recording. In addition, size and high weight make the use of such cameras more difficult.

Video technology led to further solutions.

Rotating cameras with smaller (extreme) wide-angle lenses work considerably cheaper, but they can move not record the required number of images (25/50 or 24 images per second), and these systems also have a he considerably lower image resolution.

Other video systems work with multiple cameras and mirror systems. These systems have compared to the previous ones mentioned the advantage of being able to deliver a better resolution and number of images, but they only capture one restricted image area. The images can only be reproduced using special projection systems Modification, editing or integration into a video or film cannot be realized.

The introduction of the digital video standard DV (DVGAM etc.) increases the image quality of the video systems, but solves them fundamental problems not. Subsequent processing in post-production to create videos or a binding in interactive application is not possible.

Disadvantages of the prior art

Are already listed in the individual systems.

Object of the invention

The object of the invention is to provide an easy-to-use system that already has the disadvantages systems as far as possible and can also be used more flexibly.

Solution of the task

A camera plug head, which is provided with a specially arranged row of plug connections (matched to the ver lenses / video CCDs), accommodates a number of video camera heads with a microphone. White number and position determined by the angle of view to be recorded. This allows image angles up to 360 degrees horizontally and almost 180 Record degrees vertically.

It is recorded on DV video recorders that are operated via a central control. An additional synch signal on an audio track ensures later synchronicity in further processing.

An additional video switcher allows the individual video signals of the heads to be displayed on two LCD monitors for comparison len and correct if necessary.  

The recorded data is then transferred to a computer without loss.

Using special software, the video images of the individual heads are combined to form complete (panorama) images added, from which different types of videos can then be generated depending on the requirements.

Advantages of the invention

The presented camera system is not larger than approx. 35 cm in the through even with the maximum equipment with video heads knife. The DV recorders used in the prototype are too compact in size and weight summarize transporting units. The cost of the tape material is about 2 DM per cassette at 1 hour Recording time very short.

The use of many standard components in the core area not only enables the invention to function reliably (Replacement for partial defects) but also a quick adjustment to future technical developments (higher resolution conditions with CCDs and the digital recording formats, the optics and the angle of view remain the same and thus the Ka merasteckkopf; or exchange of the digitizing and finishing unit for faster computers).

According to current calculations, the price of a basic system is one sixth to one eighth of a film camera system and, in the normally interesting image area (360 degrees horizontal, 40 degrees vertical), provides a minimally poorer physical resolution of the image than the film camera variant (film camera with scan to 4k resolution results a resulting resolution horizontally above 8400 pixels, horizontally below 12 800 pixels, vertically 700). Your own solution creates: horizontally above / below 8000 pixels vertically 700 pixels. The system can also be used as an 'electronic wide angle', as the heads can also be mounted on the camera head in other configurations. (E.g. Fig. 1, 2) The digitization costs are almost zero, since in principle only the electricity costs are incurred for the digitization time. The digitizing PC can also be used as a finishing station and, even with extreme equipment, costs no more than DM 30,000, making it several million German marks cheaper than a hand-made 70 mm film scanner.

The memory requirement for this type of processing is only around 50 megabytes per second with 15 video heads (maximum 177 megabytes per second with 49 heads) and can be easily processed on any high-quality PC.

The supposed disadvantage of the system, which arises from the possibly very high number of video heads, their adjustment is more time-consuming in comparison to conventional systems, but represents a considerable advantage of the system moderate lighting within a panorama scene can be compensated for during the recording. The flexible The arrangement of the video heads allows the rotating team to rotate the settings in two steps (e.g. Take No. 1220 degrees in one direction, then change the lighting etc., then take No. 2 at 220 degrees in the opposite direction). The An using conventional film camera systems would double the financial expenditure, if he applied the invention there is only the additional processing time that arises from the duplicate film parts (in the example it is 440 degrees) been caught).

So that these possibilities can be used to the extent possible, a process was developed with which it succeeds to compensate for the axis jumps mentioned. These are commonly caused by the fact that the pixels of the different Don't have video heads on one point. In addition, the images are rectified here, possibly color corrected and to the desired one Complete picture put together; This means that the entire recording of postprocessing or integration is available.

The commercial application concerns not only the creation of panorama videos / films in the conventional sense, but also In addition, it enables camera panning, zooming or both in digital video post production.

Thanks to the digital presentation of the material, further processing is possible, which resulted from the interactive use of the material Give video a lot more options (e.g. motion tracking for so-called HotSpots). Explanation: Motion tracking is a process of certain constant image patterns within a video / film sequence to identify and save their horizontal / vertical position values.

HotSpots are a term from the world of interactive / multimedia applications. A hot spot is an image area that is hidden in a sub-track within a picture (or a film). The user comes into this with the mouse Area, the application performs a specific function, i. H. for example, the user will branch into one offered another program area or he receives detailed information on the image area that depends on this HotSpot is covered. The combination of both methods results in a dynamic HotSpot.

These opportunities Up to now, traditional methods have not been used either for cost reasons (film) or quality reasons (video) used.

An integration in the QTVR technology from Apple (currently the most used standard) is also feasible and by me a pre-prototype already verified.

The integration into the conventional projection systems is also easier because the image sections to be used can be adapted considerably more easily to standard projectors.

Even in conventional panoramic photography, the system can be used excellently, since it can be easily combined in one moving environment can be used and thus no subsequent image retouching is required, the time-delayed recording men with a single camera are often necessary.  

Description of exemplary embodiments

1.1 Camera plug-in head: The camera plug-in head consists of a partially spherical holder, which enables modified commercially available IEEE-1394 to accommodate digital HDTV heads (or analog video heads in the pre-prototype). These heads are modified insofar as the normally existing fixed cable connections are removed and replaced by a connector. The number and arrangement of the plug connection results from the picture angles and the resolution of the camera heads used or the lenses used.
In a pre-prototype, 12 analog industrial single-chip color CCDs were used, which achieved a resulting resolution of 6000 × 460 pixels with an angular resolution of 360 degrees horizontally / 30 degrees vertically; the physical resolution of this arrangement is 3400 × 270 pixels. Another series of tests (simulation with one head) with a digital 3-CCD head resulted in a resulting resolution of 8000 × 700 pixels (with 15 camera heads) and an effective image angle of 360 degrees horizontally / 40 degrees vertically. The final distribution of the plug connections has not yet been finally determined for financial reasons.
For the structure of the head, see Fig. 1 and 2. The lenses must have fixed focal lengths (ie no zoom lenses) and are at a fixed angle to each other. The contacts in the plug head supply the heads with power and transmit the picture and sound information to the connected video recorders. The sound is only played into the left audio channel, since the channels of 'surround recording' are given by the number of heads (optionally, a permanently installed microphone system can supply 6 recorders; this corresponds to today's 'state-of-the-art') . The combined cables leave the plug head in a bundled central line. A digital signal is optionally provided for the plug connection of the heads, which feeds the assignment of the plug head (camera numbers) into the central line. Depending on the camera heads used, the motorized / electronic adjustment of the camera heads (white balance, shutter angle) is also possible via the central cable or the contacts).

1.2 Central control unit ( Fig. 3): The central control unit consists of a LANG controller (standardized for the number of recorders) (standardized protocol for the synchronous control of video recorders), which puts the DV (CAM) recorder a) in recording standby and b ) the actual recording of the recorder starts. In addition, this unit feeds short audio signals into the free channel of the recorder in a certain cycle; This ensures for later post-processing that any 'frame delays' that may be caused by the LANG control or the recorder at the start of the recording can be compensated.

1.3 Videoswitcher / LCI) monitors ( Fig. 3): These are commercially available analog components. The video switches optionally switch different inputs (single camera head) to one of two possible video outputs (LCD 1 or LCD 2 ) in order to optimally adjust the overall camera. As the number of camera heads increases, the switchers can be cascaded. Depending on the type of video heads, the video switcher is integrated between the camera plug-in head (for analog heads) and the recorders or connected to the analog output of the recorders (digital IEEE-1394 heads or digital HDTV heads).
This switcher is optional based on the invention, since the recording units (commercially available digital recorders) in most cases (likewise optional) have an integrated LCD screen. However, the switcher offers the advantage of taking up less space and making any camera directly comparable.

1.4 Summarized head control ( Fig. 3): as already mentioned in 1.1, the heads have optional motorized / electronic adjustment and are then set via this unit. The control units - if available - are included in the scope of delivery of the CCD cameras and are installed for the invention in a common housing with a central power supply; If there are a large number of heads, a switcher is also connected here, which is connected to the video switcher and switches between the individual control units in order to ensure a high level of operating safety and ease of use.

1.5 Recorder ( Fig. 3, Fig. 4): Again, these are commercially available (portable) digital video recorders that record the incoming digital (or analog) audio / video signals digitally in 4: 3 or 16: 9 format. Variants are also possible here, which are reflected in operational safety, higher resolution (video heads of the recorders) or better sampling (scanning of the color channels). The control takes place in any case with the control unit mentioned in 1.2.

1.6 Digitizing and finishing unit: The basis here is also a commercially available PC. An Apple Macintosh is currently suitable The latest design is best for this, since with these devices the IEEE-1394 interface is both hardware and in the loading drive system is integrated. The data from the digital recorder is transferred as a direct '1-to-1' copy to the hard drives) of the PC (specification of the IEEE 1394 transmission standard). The digitization of the selected sequences can be done via commercially available software (Adobe Premiere, Apple Final Gut etc.) are carried out in batch mode.

1.7 The method relevant to the invention assumes that minimal errors within image information are automatically corrected by the human brain during visual detection as long as they do not conflict with experience. Furthermore, the image angles of the individual lenses / camera units must overlap. Numerous test series have shown that this overlap must be at least 6 degrees. Fig. 5 shows on the one hand that the camera does not work in the close range of approx. 30 centimeters because some areas are not captured. In the overlap area that follows, the rays through the stylized objects (circles a, b) show the problem of the axis jump. In addition, it becomes clear (that all objects on one axis ( Fig. 6a), starting from the 'optical' center of the construction, are almost congruent in all spatial depths on the resulting image planes and that the axis jump occurs only in a minimal image area In this characteristic, this axis is initially assumed to be the later 'butt edge' of the individual images, since one lens in the given experimental setup can detect something that the lens next to it can no longer detect, 'cutting off' certain image areas in the overlap ( Fig 6 c.) d at first glance an arbitrary simplification of the subsequent overall image represents. Since, however, not perceived by an ideal lens precisely these points, this process is logical. Importantly, the image area is edge next to the shock in small areas insight allows that are not detectable with the ideal lens and thus not ent sprec reality This over-information is not perceived by the viewer as wrong or disturbing, since it appears to be correct in the context of the image (objects further away appear somewhat wider than in reality).
Since there is unfortunately no exact bet as data for almost no lens system, the detection angle must be determined or verified in a simple test setup ( Fig. 7). In addition to the optically usable area (light drop, blurring at the outer edges Fig. 7c), this experimental setup also provides the data for the distortion pattern of the lens used. An equalization grid can be generated from this pattern. This basic data plus the dimensions of the camera head serve as the first basis for the basic configuration of a computer model. Depending on the lens / camera system, 5 to 7 camera heads must be included in this virtual test setup ( Fig. 8).
The future camera plug head is pre-designed in the computer and placed in a virtual test setup. The grids, which are offset in depth, provide visual information about the axis jumps that occur. Since 'computer lenses' are not affected by the optical distortion of real lenses (simple vanishing point calculation), a certain 'angular' pattern results after the individual images have been put together at the previously determined abutting edges ( Fig. 9). The intersection points of the grids ( Fig. 9a) form support points of a family of Bezier curves that are centered one above the other in the vertical image axis ( Fig. 10). The central part of this grid is used as a distortion grid on all individual images ( Fig. 11) and, when applied to the individual images, results in a homogeneous virtual wide-angle image that still allows the vertical axes of the objects to be vertical ( Fig. 12).
If you now combine the two distortion patterns into one, this results in a process that first turns the individual video image into an 'ideal image' and then fits this into the second pattern.
However, it must be seen in a restrictive manner that real images cannot simply be put together at the abutting edges, but require a transition area (soft screen) (Figure (13 and) 14). This is where the problem of the axis jump comes into play in this transition area. In order to compensate for the image errors in this area, the image in the edge area is 'spatially bent backwards', so to speak. The averaging of the values obtained from the virtual test setup (or by vanishing point calculations) leads to minimal distortion (for clarification Fig. 15) in the overlap area. This distortion depends on the lens used and the area of overlap, but is no longer visually perceptible (even no longer necessary if there is a large overlap). In the following, the averaging of the values for several depth grids is carried out, depending on the increasing spatial distance of the grids, the distance to the abutting edge becomes smaller and smaller and a curve function results from the iterative application of the distortion. This solves the problem for the camera ring in the equator of the plug head.
The pictures of a second / third ring mounted above or below the first can basically be put together using the same procedure. However, another equalization process must be inserted. Since all heads capture part of a sphere, the center of which is in the center of the head, the resulting image is part of a sphere development. The centers of the images lie on a segment of a circle, the radius of which results from the lens geometry (and thus vertical angle arrangement of the series) and the resolution of the CCD chip. The image composed in this way is then converted from a polar coordinate to a standard coordinate system and added to the first row.

1.8 Using this procedure, software assigns a specific camera to each camera according to its position Pattern too. The thus corrected or distorted images can then be seamlessly juxtaposed, added together and as one individual images are output. The output as single images in TGA or TIFF format is currently necessary because there is no computer format can play such large films. In many cases, the images must be used for further processing in video post production can also be split.

1.8.1 Computer-based interactive applications can, however, use directly playable films in standard video resolutions or smaller resolutions. To this end, a viewing window is passed over the panorama image sequence in small steps from left to right (or vice versa). Each step in itself creates a video sequence. All sequences created in this way are combined to form a continuous overall sequence. This creates an image matrix of X × Y individual images, where X indicates the number of steps for moving the window and Y the number of images within the panorama video sequence. This matrix can be defined under Apple QuickTime® as a so-called VR object. Although this application completely reverses the purpose of this QuickTime application, it enables interactive viewing of the panorama film (this special edition was already implemented by me under QuickTime®).
The previous application of standard software applications (which allow motion tracking) to the resulting panorama video sequence enables HotSpot data to be transferred to one or more alpha channels. If the method described above is now used, this information is also transferred to the matrix and thus allows additional interactivity when viewing the panorama.
Any necessary color, brightness corrections and register shifts (caused by the play of the camera plug heads) and synchronization of the individual video sequences can also be set within the software using known algorithms.

1.9 The prototype of this software was created by scripting and connecting individual components of commercially available software pa kete realized by me. Image sequences were digitized for the real experimental setup using Adobe Premiere (these sequences zen were added to a still image to increase the image quality). The pictures taken were traced and averaged in Adobe Illustrator. The resulting Bezier curves were based on their mean point developed a uniform grid. These two grids formed the beginning and end of a 'morphing process' in Avid ElasticReality. Construction of the 'virtual' model (and also the first pre-prototype) was carried out in Yonowat Amapi and Play ElectricImage for the implementation of the 3d cameras. The 'grid test series' were then generated in the latter. The so he generated frames were assembled and masked in Adobe AfterEffects (temporary clipping of the overlap areas), the resulting individual images were used as templates within Adobe Illustrator for creation the Bezier curves for the distortion of the computer images, also from these Bezier curves an averaged family of curves (for the higher accuracy). This family of curves was again imported into Adobe AfterEffects and with the undistorted and unmasked center image masked, so that the general distortion pattern resulted. The distortion pattern served as a hint Background for generating a 'mesh warp' on an empty image with the identical size of the calculated 3D image. This Warp was now copied to all masked images, which resulted in the 'orthogonal wide angle'. Exchange of rech ner-3d images with such computer images that have more distant space grids result in the straight family (within the Overlap / crossfade area), which in turn was the basis for the creation of a second 'MeshWarp'. This second Warp was copied over the first one. The equalization initially made in ElasticReality has now been changed to an undistorted one Raster image applied. This distorted pattern served as a template for another within AfterEffects 'MeshWarp', which was applied to the following video sequences. All of the distortions were significant in one higher resolution than the video resolution generated in order to achieve high accuracy. The later color, bright Speed and register corrections were also carried out in After Effects.

Claims (6)

1. Device for capturing a panoramic video with several camera heads Fig. 3 (a), characterized in that a plug head Fig. 3 (a) holds several video camera heads Fig. 3 (b). 2. A method from a sum of individual videos, which are created with the device of claim 1 or 3, an overall video to create or image, characterized in that the resulting axis jumps, distortions, color differences be compared and these images are merged into one another without visible transitions. 3. The method according to claim 2, characterized in that one of the number and orientation of the video camera heads dependent field of view in a plane up to 360 degrees in the X-Y direction. Because of their digital copy This layer can contain invisible sub-layers with information on how to control the resulting film or view control of other programs included. 4. The device according to claim 1, characterized in that the camera heads can form an 'electronic wide angle' Fig. 1st 5. Device according to one of the preceding claims, characterized in that an electronic adjustment of the camera heads is provided Fig. 3 (f). 6. Device and method for controlling and synchronizing video recorders with which picture sequences according to claim 1 or 3 are recorded, characterized in that a free audio track of the tape is an additional absolute Synch signal picks up.