NO20130840A1 - Camera based, multitouch interaction and lighting system as well as method - Google Patents

Abstract

Device, system and method of control and interaction within an interaction volume (1) within a height above the coordinate plane (12) of a computer, such as a computer monitor, interactive whiteboard, horizontal interaction surface, video / web conference system, a document camera, a rear projection screen, a digital sign surface, a television screen or a gaming device, to provide pointing, gliding, selection, banking, gesturing, scaling, drawing, writing and erasing using one or more interaction objects (2), such as fingers, hands, feet and other objects such as pens, brushes, brushes and even more specialized tools. The device and method can be used in a system together with, or also integrated in, data projectors (3) of all types and their attachments / holders (4), and used with flat screens (12) (such as LCD, plasma, OLED, rear projection screens, etc.) to make such viewing systems interactive. The device consists of a single camera (5) covering the interaction volume (1) from either a very short distance or from a greater distance, to determine the lateral positions (X, Y) and also capture the position of the one or more interaction objects ( 2).

Description

Technical area

The present invention relates to camera-based interactive multi-touch systems which, for example, use camera-based input devices and visual and/or infrared lighting to track objects within an area/room, for example to track one or more fingers or a pen for human interaction with a computer; in that the system enables the determination of a two-dimensional position within an area and a height above a surface of the area to provide current two-dimensional input coordinates and to accurately distinguish between current interaction states such as "inactive" (no tracking), "hovering" (tracking although there is no touch, sometimes also labeled "in the area") and "touch". The present invention also relates to multi-touch input devices and interfaces to enable both pen and finger touch input, and is also capable of dealing with multiple objects at the same time, for example a multi-touch input device to a computer. The invention further relates to methods for gesture input using three-dimensionally based input devices and thereby capturing the position of, for example, a hand or a finger, and sequences of these can be recognized as gesture commands and/or position and orientation inputs for three-dimensional control.

Background for the invention

Camera-based tracking of objects for human computer interaction, especially tracking of the hands and fingers, has been the subject of scientific, industrial and commercial interest for decades. Overviews of what has been achieved in this computationally intensive area are provided by Pavlovic et al., IEEE Trans. Pattern Analysis and Machine Intelligence, vol 19, No.7, pp. 677-695, 1997 and by Zhou et al., IEEE Int. Symposium on Mixed and Augmented Reality, pages 193-202, 2008.1 many of the reported techniques, the object is observed from several different viewpoints by one or more cameras to reduce sensitivity to occlusions and for robust tracking and gesture interpretation.

For single-camera-based tracking of finger touch and finger or hand gestures, features such as shadows, contours, texture, silhouette, and image gradients of these objects and even their mirror image reflected from a blank screen surface are extracted and used to update the various model-based tracking the systems to calculate the finger or hand position and to detect, for example, finger touch in real time.

As an example of cunning feature extraction, US2010/0066675A1 describes touch screen system with single camera imaging and feature extraction based on the observation that the shadow of a finger illuminated by a lateral illumination source is eventually eclipsed by the finger upon touching the screen, so that the shadow resembles a finger when not touching, while the shadow is sufficiently reduced when the finger touches the surface so that touch can be determined. However, the independent patent claim is pre-empted by the public scientific article from 2005 by the inventor Andrew D. Wilson (ACM Proe. UIST 2005, pages 83-92).

WO9940562 (A1), US 006100538A and US2010188370 (A1) describe in principle object tracking systems for finger touch or pen where at least two camera viewing points are arranged at the edge of the coordinate plane to determine the coordinates of the object by triangulation.

WO9940562 (A1) describes a system for detecting pen and finger touch in front of a computer screen using a single camera and by means of a periscope-like optical system consisting of one or more flat mirrors, recording two images of the screen facing sideways into the volume immediately in front of the screen to determine the coordinates of the pen or finger and the distance to the screen.

US006100538A describes a digitizing device for determining a position of a pointing object which projects a light and is placed on a coordinate plane, and a detector arranged on the periphery of the coordinate plane, preferably a pair of linear image sensors, has a field of view covering the coordinate plane, and a collimator is arranged to limit the height of the field of view of the detector, and the detector can receive only a parallel component of light projected from the pointing object, substantially parallel to the coordinate plane, and a screen is arranged to block noise light, other than the projected light, from entering into the limited field of view of the detector, and a processor is provided to calculate the coordinates representing the position of the pointing object.

US2010188370 (A1) describes a camera-based touch system that includes at least two cameras having overlapping fields, located along the periphery and typically in corners of the touch surface to detect the position of the pointer by means of triangulation, and to detect the pointer touch and pointer hover over the touch surface.

JP63292222 and US2008152192 (A1) mainly use a camera placed far away from the object and use one or more mirrors within the camera's field of view to observe the object from different viewing angles and directions which are mainly perpendicular to the camera axis, to facilitate the detection of the object's position.

JP63292222 utilizes a single camera at a distance from a writing surface and two flat, narrow mirrors along the periphery of the writing surface in each of the two directions X and Y inclined to the surface to provide alternative viewpoints of the pointing device, making it possible to provide X and The Y coordinate separately by capturing and analyzing the two mirror areas along the writing surface area.

US2008152192 (A1) describes a system for three-dimensional monitoring and analysis of movement-related behavior of test objects, namely fish and animals. It comprises a real camera and at least one virtual camera realized by using at least one plane mirror within the field of view of the real camera, which represents at least one alternative point of view that can be analyzed in one or more areas of the captured camera image in order to analyze the movement behavior to the test objects.

In a published international PCT patent application number WO2005/034027(A1)

(Smart Technologies Inc.), a device for detecting a pointer within an area of interest is described. The device includes a first reflective element extending along a first side of the area of interest, and arranged to reflect light towards the area of interest. The device also includes a second reflective element extending along a second side of the area of interest, which is also arranged to reflect light towards the area of interest. The second side is connected to the first side to define a first corner of the device. A non-reflective region substantially in a plane of at least one of the first and second reflective elements is adjacent to the first corner. At least one imaging device is arranged to capture images of the first region of interest, including reflections from the first and second reflective elements, to determine a position of the pointer within the region of interest.

In a published Japanese patent application number JP63292222(A) (Mitsubishi Electric Corp.), an optical system adapted to detect a coordinate position of a point indicator is described. The optical system works by forming an image near an upper plane relative to a corresponding original object. Also included is a processing arrangement for processing an image signal obtained from an image sensing device arranged near the upper plane. The image sensing device more particularly senses an image coverage area that includes a read origin. Also included is a reflective X-direction mirror and a reflective Y-direction mirror. The point indicator, for example a white pen, is sensed directly by the image sensing device and also via reflection from the mirrors, allowing the image signal to determine a spatial position of the pointer indicator within the optical system.

Furthermore, in a published Japanese patent number JP4484796(B2) (Canon KK), there is described a coordinate input device for accurately detecting a coordinate input thereto. The device includes a number of sensor units arranged around the coordinate input surface, where each of the sensor units includes a projection part for projecting light radiation on the coordinate input surface and a light-receiving part for receiving incoming light at the sensor unit. The device also includes a number of recursive reflection parts which provide recursively reflected incident light provided on a circumference of the coordinate input surface. The device is arranged to calculate a coordinate for the pointing position of a pointer based on the light quantity distribution which includes light shielding areas obtained from the number of sensor units. A three-dimensional light-shielding detection area regarding the number of sensor units has a regular three-dimensional shape corresponding to the coordinate input surface. The three-dimensional light-shielding detection area is further defined as a three-dimensional area in which a change in the height direction position of the pointer is detected by a change rate of light intensity as detected by the number of sensor units.

In a Japanese patent number JP4033802(B2) (Advanced Telecomm Research Institute), a large touch panel system is described which enables touch input of information. The system includes a screen of a plastic material which is irradiated by an infrared source during operation from a front side of the screen. During operation, a person manually touches the screen with their hand. A camera in the system also photographs a backside of the screen via a mirror to generate photographic data that is delivered to a computer. Based on the photographic data, a shaded area resulting from the person's hand interrupting the infrared radiation can be detected by the computer by processing the photographic data. When the shaded area has a spatial extent corresponding to the size of, for example, a hand, the coordinates of the shaded area are determined to derive a measure of a spatial position of the person's hand relative to the screen.

It is generally important that the user's intentions and commands are correctly recognized in human/machine interaction systems. The accuracy of X and Y in the coordinate plane may or may not be important. This depends on the application. Finger touch systems are therefore attractive where modest accuracy is required for, for example, moving or selecting graphical objects or entering menus, while a pointer or pen is preferred when the highest accuracy is required, for example fine writing or drawing or handling all details and objects in CAD programs. In a finger-based system, therefore, feature extraction and robust heuristics for determining the finger's coordinates may be sufficient, based on a two-dimensional image from a single camera.

However, for all application types, high accuracy regarding the detection of finger or pen touch is of the utmost importance and must never fail because the user may then lose control over the application. A high and constant detection quality of the touch state is therefore necessary in any position in the coordinate plane. Furthermore, the detection method should not be sensitive to variations in finger size, skin colour, ambient light conditions, screen light etc., and the detection should be fast. A good user interaction is therefore designed to ensure high quality, high robustness and high speed for the finger/pen touch detection even if the coordinate resolution accuracy is modest, and the best system will be able to provide the physical height of the object with constant scaling over the entire the coordinate plane, thereby determining both hover and touch state uniformly across the coordinate plane and without any user-dependent behavior or delay penalty.

For determining position, scaling is not that important. The relationship to the distance between different features observed within a single image can, for example, be sufficient to determine that the object in question is a hand with, for example, a straight thumb and a straight index finger, while the other fingers are hidden. It is not important whether it is a large hand of a man or a small hand of a child, or whether it is large because it is close to the camera lens or small because it is further away. By tracking the relative movements and accompanying posture types that can be determined from frame to frame, such sequences can be interpreted as hand gesture commands that are to some extent incorporated into user interfaces for computers, mobile devices and integrated systems.

There is great interest in interaction systems that use pen, touch or both (dual mode systems) for education, collaboration and meetings. Operating systems and graphical user interfaces prepared for dual-mode multi-touch and multi-pen input distinguish between touch, pen and mouse input, and therefore dual-mode input devices must report multi-touch, multi-pen and mouse information simultaneously to the computer. Several new interaction platforms also allow single pen or finger gesture control and/or even hand gesture based information.

There is particularly great global interest in interactive whiteboards and whiteboards for use in teaching both in ordinary classrooms and in larger teaching halls. Such whiteboards are eventually also found in meeting rooms, rooms for video conferences and work rooms. The images on the interactive whiteboard coordinate plane can be generated as a projected image from a long-throw or short-throw projector or a flat screen such as an LCD screen, a plasma screen, an OLED screen or a backlight projection system. It is important that the touch and/or pen input device can be used with all types of display technologies without reducing image quality or wearing out the equipment. It is also important that the input technology can be easily adapted to different screens, projectors and display devices with low prices and effort.

New interactive whiteboards are usually equipped with short-throw projectors, namely projectors with an ultra-wide-angle lens placed a short distance above the screen. With the help of this solution, the user will be less disturbed by light in their eyes and will tend to cast less shadow on the screen, and the projector can be mounted directly on the wall together with the board. An ideal input device for pen and touch in connection with such short-range systems should therefore be integrated into or fixed along the wall projector, or fixed to the projector's wall mounting to make the installation simple and robust.

In lecture halls, very long, interactive whiteboards and interaction spaces are needed, and these interaction surfaces should provide touch, pen and gesture control. On large format screens, joysticks and laser pointers are often necessary to attract the audience's attention. The preferred input technology should suit all such different needs, i.e. also accept styluses and lasers as a user input tool, and be tolerant of and adaptable to different display formats.

Also, flat screen technologies may need touch and/or pen operation, single pen and/or touch gesture interaction, and finally hand gesture control. Touch-sensitive films placed on top of a flat screen cannot detect hovering or gestures in the air. Purely electromagnetic sensing systems behind a flat screen cannot detect finger touch or finger gestures, only electromagnetic pen operation is possible. Some types of flat screen technologies, especially OLED screens, can be transparent, and thus camera-based technologies can be used for gesture control through the screen. If dual-mode input systems incorporating hover and gestures continue to become more important and standardized to provide an efficient and natural user interface, optical-based input systems are likely to be preferred also for interactive flat screens instead of capacitive or resistive films or electromagnetic-based solutions. The preferred technology for input devices should therefore be optically based and should be suitable for adaptation to both conventional flat screens (LCD, plasma, LED) and transparent flat screens such as OLED and rear projection screens.

Input devices should not be sensitive to light sources such as daylight, room lighting, light from the projector or display screen, etc. The input devices should also not be sensitive to near-infrared radiation from sunlight, artificial light or from remote control devices and the like that use diodes that emit near-infrared light for communication .

The input devices should also exhibit a high update rate for coordinates and provide low latency for the best possible user experience.

The input devices should preferably be suitable for existing infrastructure to, for example, upgrade an already installed pen-based interactive

whiteboard model to also enable finger touch and hand gesture control, or to upgrade a meeting or teaching room already equipped with an installed projector or flat screen to become interactive using the input device itself.

In some scenarios, input technology can also be used without interactive feedback on the writing surface itself, for example by accurately capturing the strokes of the chalk and sponge on a traditional whiteboard and recognizing hand gestures to control the computer, or by capturing the normal use of pen and paper ( including swipes) and simple gestures for controlling the computer; or by capturing the user's information by completing a paper form or questionnaire that includes his/her signature, while the result is stored in a computer and the input or other interpretation of the input is displayed by means of the normal computer screen or by means of an attached display device or a projector for reference for the user and audience. This means that the input device should be possible to use alone or separated from expensive display technology in cases where this type of infrastructure is not available or necessary.

In the same way that interactive whiteboards are replacing traditional chalk and blackboards in teaching, new interaction spaces are emerging in other areas. Multi-user interactive vertical and horizontal surfaces are being introduced in conference rooms and control rooms, museums and exhibitions.

Interactive places including interactive gesture tables are created in bars, casinos, cafes and shops to enable guests to choose from a menu, order and pay as well as be entertained by, for example, playing computer games, searching the internet or reading news . Interactive rooms will be utilized within digital signage using flat screens or projector screens with digital content that can be changed dynamically, not only in a predetermined sequence from the content provider, but changed due to user input from touch and gesture control to thereby make the display of the sign or the content even more flexible, informative and user-friendly. Touch and gesture control input devices for use in interactive content display should work well through thick vandal-proof windows and work well on all types of surfaces and flat screens with easy installation, to be suitable for installation and use in public and commercial areas indoors and outdoors .

Summary of the invention

The present invention relates to a device, a system and a method for an input device in man-machine communication for tracking the position of an object within a coordinate plane; for the detection of hover and/or contact states within a volume located at the coordinate plane within a given height range; and/or for recognition of the object's position,

which has

a camera to take a picture using visible light and/or near infrared light,

a mirror arrangement arranged at the coordinate plane, and

a unit of calculation

wherein the camera's field of view includes both the coordinate plane and the volume above it and the mirror arrangement, wherein the mirror arrangement comprises at least one axially offset, concave, substantially parabolic element with its axis parallel to the coordinate plane and its focal point at the camera's entrance pupil to provide a constant magnification of the height dimension of the volume along its axis, so that the object's coordinates and/or its hover and/or touch state and/or position characteristics can be calculated based on a single image using the calculation unit, and/or the object's movement and/or the object's gestures can be calculated based on a sequence of images using the calculation unit.

The camera comprises a CCD or a CMOS imaging chip or the like, and a lens or the like with a field of view large enough to include the coordinate plane, the volume and the mirror arrangement, and with a sufficient optical image quality so that the relevant wavelength ranges are adapted to the relevant image resolution - the chip.

The present invention has at least one mirror arrangement which comprises one or more axially displaced, mainly parabolic elements distributed at the surface on the outside of the periphery of the coordinate plane. Each such off-axis, essentially parabolic element has its focal point in the camera's entrance pupil and its axis parallel to the surface. The property of each off-axis, essentially parabolic element is to collimate a set of parallel light rays, parallel to the surface, emanating from the object when the object is inside or partially inside the volume. This property ensures that the measurement of actual height, namely the distance between the object in relation to the surface, has constant magnification/scaling for this element, and can be easily determined by locally analyzing the image area covering this mirror element. The mirror arrangement or arrangements must also be arranged to ensure that the different sets of parallel light rays from each parabolic element together cover the entire volume above the coordinate plane without dead zones, so that there will always be at least one mirror element covering the object when the object is within the volume, and which can be observed from the camera's point of view with a sufficient number of pixels so that the calculation unit can determine the object's current height and can thus determine the object's touch state and/or hover state. The camera may also be equipped with standard or custom bifocal lenses or the like to magnify the mirror arrangement at the expense of its surroundings to thereby increase the resolution of the image of the mirror arrangement in the pixel array of the camera sensor to a sufficient

resolution level for accurate height determination.

Although the one or more off-axis, substantially parabolic mirror elements are distributed at the surface of the coordinate plane arranged to sense the height of the object above the plane, additional curved or planar mirror elements may be located further outside the off-axis, substantially parabolic mirror elements to provide spatial information about the scene when these latter mirrors are observed from the camera's point of view.

There will be further restrictions on the placement and orientation of the various off-axis, mainly parabolic mirror elements in the case where there are obstacles to a straight line of sight between the camera's entrance pupil and the surface, for example due to the mechanical shape of the projector, the size of the wall mount, etc. .In a preferred embodiment, the set of possible areas for placement of the axially displaced, substantially parabolic elements outside these obstacles is recorded, and then the location of each element is selected from this set and assigned an axial direction along the surface to distribute the beam bundles sufficiently evenly across the coordinate plane without dead spots or shadows, while the resulting shape of the mirror arrangement or arrangements should be smooth and/or well adapted to, for example, the wall mounting mechanisms and/or projector shape in terms of ease of manufacture, ease of assembly and good aesthetic appearance.

In preferred embodiments, the mirror arrangement is a set of axis-shifted, mainly parabolic elements distributed in a semicircle around the wall or table mount for the input device. The smaller the radius of the semicircle, the more wide-angled the resulting optics will be, making the image of the object's width more dependent on the distance between the object and the mirror surface. If this radius is too small, it will be difficult to measure the height of the object at a great distance from the mirror since the image in the width direction is reduced too much even though the height dimension has constant magnification scaling regardless of the distance from the object to the mirror. An appropriate choice for the radius can be found for the given image and lens resolution, mirror surface quality and sensor light budget, as well as the mounting and coordinate plane geometries.

In an alternative preferred embodiment, the axially displaced, mainly parabolic mirror elements are arranged in at least one straight casting along at least one of the circumferential edges of the coordinate plane. This is a favorable location to ensure good observability and distribution of the optical rays coming from the object, it can be easily manufactured and mounted and can result in a good aesthetic appearance.

In some alternative embodiments, the mirror arrangement may be shaped as a mosaic of small off-axis, substantially parabolic segments, each arranged to collect parallel optical rays originating from the object, at different positions and heights, but where the structuring of the direction, location and height of the segments is optimized to cover the volume above the coordinate plane in the most efficient way for a space-constrained or shape-constrained mirror arrangement, or to find the most efficient shape of the mirror arrangement for detection coverage of a given smallest object size. By exploiting mosaic structures, it is possible to find arrangements that provide optimal observation, less shadowing and a good mapping from mirror to pixel at the cost of more design optimization efforts and image decoding complexity.

The mirror arrangement can be produced directly in metal using various fabrication technologies such as milling, turning, three-dimensional laser engraving, grinding or EDM. In order to make the manufacture suitable for large volumes and cheap production, injection molding of plastic materials and metal coatings deposited on plastic can preferably be used and will also reduce the weight of the mirror arrangement. For surfaces with higher quality and precision, metallized glass substrates of various types can be used. For mirrors with lower volumes and lower quality, thermoforming and vacuum forming of mirror-like metallized plastic films glued to a substrate may be possible when the radius of curvature is large. Punching and shaping of polished sheet metal can also be used to make the mirrors of a quality sufficient for some preferred embodiments of the present invention.

The mirror arrangement can also be produced by utilizing internal total reflection in materials such as plastic or glass.

The mirror arrangement can also be produced by utilizing internal total reflection in plastic or glass materials in combination with metal coating to protect and extend the mirror function (for angles smaller than the critical angle at which total internal reflection occurs).

In some preferred embodiments, the mirror arrangement based on total internal reflection may be made of Fresnel-based segments. In some preferred embodiments, the mirror arrangement may be made of a combination of plane mirror segments at given angles and a plastic lens or a plastic Fresnel lens to provide the required resultant curvature for the off-axis substantially parabolic function.

In some preferred embodiments, the off-axis, substantially parabolic function may be realized using a lens or Fresnel lens such as those manufactured for solar energy applications.

In some preferred embodiments, the one or more mirror arrangements may be covered by a layer of plastic and/or a special coating that selectively stops or lets light pass through within given wavelength ranges. The outer layer or housing may then appear to be homogeneous with, for example, a constant dark brown color observed by the user and the public in visible light, while the mirror behind the coating is fully functional in the near infrared light within given wavelength ranges from the imaging camera.

In some embodiments, the ambient light and/or the light from the screen (that is, light from the projector or from the flat screen, respectively) can be used to illuminate the object.

In some preferred embodiments, an illumination arrangement in the visible and/or near infrared range may be included to illuminate the object directly and/or indirectly by means of the mirror arrangement.

In some preferred embodiments, the lighting arrangement may be controlled by an on/off switch to selectively turn the lighting on or off for different images.

In some preferred embodiments, the illumination source in the illumination arrangement flashes within the active exposure period of the camera to freeze the motions related to moving objects.

In some preferred embodiments, the illumination arrangement is located near the entrance pupil of the camera, namely near the focal point of the parabolic elements, to illuminate the object through the mirror arrangement to thereby spread the light in the volume located at the coordinate plane at a given height and with rays that are mainly parallel to the plane.

In some preferred embodiments, an illumination arrangement is located near the entrance pupil of the camera to directly illuminate the object.

In some preferred embodiments, there is a common illumination arrangement for illumination of the object through the mirror arrangement and for direct illumination of the object.

In some embodiments, there are separate illumination arrangements for illumination of the object through the mirror arrangement and for direct illumination of the object.

In some embodiments, there are separate mirror element arrangements for illumination and observation so that the mirrors in the arrangement for observing the object's height above the coordinate plane are less exposed to the illumination arrangement itself, thereby reducing unwanted reflections from the optical interfaces and thereby increasing the signal/noise ratio in the measurements.

In some embodiments, on/off control of the illumination arrangement for illuminating the object through the mirror and on/off control of the illumination arrangement for illumination of the object are directly separated so that object illumination from the illumination arrangements can be selectively turned on and off for the different images to provide better detection of the object, for example to provide contours around the object in side lighting.

In some embodiments, the lighting arrangements also include visible light, for example multi-coloured light-emitting diodes with on/off control, so that the object, for example a finger, can be illuminated with colored light, for example green through the mirror arrangement, thereby signaling to the user that the selected ink color for example is green. In the same way, flashing red can be signaled to the user on their finger as a kind of alarm without being observed by the spectators, etc.

In some preferred embodiments, the camera includes an optical filter to block unwanted light, namely the light from the flat screen or projector screen and/or ambient light, while allowing light of the same wavelength range as the illumination to pass through.

In some preferred embodiments, the camera comprises one or more selectable optical filters that can selectively block or pass light in different wavelength ranges, thereby, for example for some images, allowing light of the same wavelength range as the illumination to pass through, while for other images allowing, for example, only ordinary light to pass through in order to be able to capture the images from the projector or flat screen.

In some preferred embodiments, the present invention can be combined with the inventions in WO2001NO00369 / US7083100B2 and/or WO2006135241A1 / US2009040195A1 with objects that are equipped with patterns that can be observed either directly or through the off-axis, mainly parabolic elements or both with a given wavelength range on their surface and/or inside its body and/or projected on the screen, as a means for more accurate tracking and/or for identification of the object and/or for detection of the state of various user interaction controls, such as buttons, etc., as according to the above inventions can change the observable patterns. The proximity of the object to the surface or the proximity between different internal components of the object can also be observed by combining the present invention with the optical proximity detector described in WO2005050130 / US7339684B2.1 such preferred embodiments, the observation of such patterns and/or proximity information can especially be done through the present invention's mirror arrangement or mirror arrangements to thereby provide constant magnification of this optical information over the entire coordinate plane.

In some further preferred embodiments, the above-mentioned patterns are made on the object by applying well-known retroreflective principles for at least a given wavelength range to take advantage of the fact that the illumination arrangements are located close to the entrance pupil of the camera so that the retroreflective property of the object will ensure high intensity of the the direct observation and/or the observation through the one or more mirror arrangements.

In some preferred embodiments of the present invention, a simple computer-based calibration procedure can be used to find an accurate mapping of the coordinate plane to the display coordinates. A common way is to let the calibration procedure be user-assisted by showing crosses at several points on the screen, while a manual pen or finger touch is required to find the mapping, namely the transfer matrix.

In some preferred embodiments of the present invention, a computer program can output images to the display device with, for example, patterns used to identify and track objects as in WO2001NO00369 / US7083100B2 and/or WO2006135241A1 / US2009040195A1, which can be automatically recognized by the camera to find the transformation matrix to map the coordinate plane to view coordinates. Since the present invention is the imaging of two different viewing angles for an object located in the volume above the coordinate plane, some preferred embodiments of the present invention may include a calibration and control program also for the height dimension, i.e. to control and/or adjust the thresholds correctly for accurate touch and hover by including a test object that can be observed directly and through the mirrors respectively. Semi-transparent, three-dimensional pattern objects may be illuminated by the display device as part of this calibration procedure. As an illustrative example, a semi-transparent cylindrical test object with, for example, some opaque bands along its surface and/or opaque objects inside its volume, is placed in some positions on the display device which is highlighted in circular areas one by one using the calibration program. The display device will illuminate the semi-transparent test object when it is placed over these small circular areas so that it can be seen directly by the camera and can be seen from the side through the mirror arrangement, according to the present invention. The test object can have opaque and transparent details that are dimensioned to be observable in the camera's two viewing angles according to the present invention in order to identify and distinguish different test objects; to calibrate and establish the mapping from coordinate plane coordinates to display device coordinates; and/or to calibrate or regulate the altimetry, including the determination and/or of the thresholds for the touch and hover conditions for a given installation.

In some preferred embodiments, the mirror arrangement and/or the projector holder and/or the screen holders and/or the writing surface may have optical patterns for accurate object positioning in the scene, as described in WO2001NO00369 / US7083100B2. This can greatly simplify the assembly and calibration procedure, and the calibration can be done internally using the calculation unit in the input device without manual calibration steps or external computer programs.

The purpose of the present invention is to provide positional information in the X and Y directions, as well as information about touch and hover (the Z direction representing information about user action) from the user in a man-machine interface that typically also includes, but does not is limited to, an interactive display device.

It is further an object of the present invention to be used for advanced multi-touch interaction which is used in human interface devices for computers and other electronic equipment. The fine details of user interaction including precise touch control, hand posture and gestures can be captured using a combination of direct observation and observation through an axis-shifted, mainly parabolic mirror arrangement. By using flashing lighting directly or through the mirror arrangement, all movement can be frozen to avoid smearing of the camera images. In some preferred embodiments of the present invention, the lighting can also be provided with separate optics, thereby removing reflections involved in illumination and observation done simultaneously through the same optics, thereby improving the signal/noise ratio.

In some further embodiments of the present invention, illumination devices with near infrared light are used. The camera may also have an optical filter that blocks visible light, etc., and only allows near-infrared light to pass. In such embodiments, the invention will be less sensitive to other light sources such as daylight, room lighting, light from the projector, screen light, etc.

It is an advantage of the present invention that the magnification of the interaction objects is constant for all distances for a given segment. This involves simple image processing and a very accurate system over large surfaces. The purpose of this invention is to create a very robust and accurate touch and hover detection system.

It is also an advantage of the present invention that it becomes possible to include it in front and rear light projection systems on walls and on boards, and the present invention can be either integrated into new equipment or retrofitted into existing equipment to make such systems interactive.

It is a further advantage that the present invention can be mounted on or integrated into wall holders for projectors or screen holders (LCD, OLED, etc.).

In some alternative embodiments of the present invention, for very advanced interaction spaces, the use of bifocal camera lenses can improve resolution by enlarging the image around the mirror arrangement to obtain even more accurate touch and height information. Alternatively, the lens optics may be separate for direct viewing and viewing through the off-axis, essentially parabolic mirror arrangement to miniaturize the equipment, reduce costs, and simplify installation. This can be achieved by exploiting available and inexpensive CMOS image sensor technologies that provide full exposure synchronization and streaming of a pair of images from two separate sensors using an interconnected high-speed serial connection, then using the lens optics best suited for the two separate images, and then perform the same calculations on the image pair using the calculation unit. The method for speed increase which is described for the present invention will also be able to be used in such a double sensor/lens configuration.

The present invention can utilize inexpensive CCD or CMOS camera technology and inexpensive near-infrared LEDs and optics that are easy and cheap to manufacture, and available signal processing integrated circuits that are easy to program for the application at hand. The present invention is therefore easy to implement in large production volumes.

In some scenarios, the present invention can also, for example, determine hand positions as a second interaction object within the camera's field of view, but not necessarily within the defined interaction volume, where the position of at least one first object is determined so that the position of the second object can provide additional information in the human interaction with the computer.

The method, which is based on observing the object using the off-axis, essentially parabolic mirror arrangement, explicitly provides the height Z above the interaction surface, synonymous with hover level information. By only performing simple edge detection over those camera pixels that represent the various off-axis, mainly parabolic mirror elements, it becomes possible to determine the presence and height of an interaction object. One can also use different image processing methods to detect the relevant changes in the image areas of the mirror elements, such as, for example, subtraction of a reference image, and also absolute differences in the image, as well as normalization, threshold control (that is, comparison with one or multiple threshold values) to, for example, find a binary representation that can easily be further processed to find candidate objects using speckle detection algorithms or template matching techniques. The candidate objects can be located both in the mirror image that looks along the interaction surface, and in the direct image, namely the image of the interaction surface itself. The so-called correspondence problem, namely where corresponding image information from two different viewing angles must be identified, is generally a very complex problem, and is a key problem in stereographic and artificial three-dimensional viewing systems. By utilizing the off-axis, mainly parabolic mirrors, information about the height (Z) is explicitly and linearly represented without any perspective distortion as a function of the object's position (X, Y) in the interaction volume. The correspondence problem for the present invention is therefore reduced in complexity compared to a general case, but can be further simplified using the method described below.

The method which is based on observation of the object by means of the off-axis, mainly parabolic mirror arrangement provides, as already discussed, explicit information about hover level. The method also makes it possible to find the interaction objects faster by exploiting a characteristic that the image through the axis-shifted, mainly parabolic mirror element is a view along the interaction volume in a particular direction, which means that several object positions are assigned to the one simple mirror element or a group of such mirror elements and can be observed by the camera which has a low number of pixels. For the initial search for where the object is located, the imaging process of the limited camera pixel area related to the mirror arrangement will easily find the height of the object and the direction where the object is located. In the aforementioned mirror arrangement where the axially displaced, mainly parabolic elements are distributed in a semicircle, then the height (Z) and the azimuth angle (AZ) representing the direction of the object can be found directly, and a trajectory of positions for the candidate object in the interaction volume can be determined. This path can be transformed into a path in the image sensor group by, for example, a look-up table, and the first instance of the object can then be sought by, for example, an edge-detecting algorithm run along this path. This approach represents an efficient search procedure for finding the one or more objects in the image with high computational speed compared to a full two-dimensional search of the image sensor array for the entire interaction surface.

Furthermore, a redundancy method can be utilized in connection with the present invention to find the position, hover level and touch of the one or more objects even when the direct image of the object or objects is blocked in the direct camera angle, by utilizing two or more mirrors. The user can occasionally and unintentionally hide the pen, his fingers or his hand during an interaction session by, for example, his other hand or his head seen from the direct camera point of view, while two mirrors along the interaction surface can follow the objects, determine their height, find their touch- and hover state and calculate their positions by triangulation of the azimuth angles of the objects observed in the mirrors with a given base length.

By tracking the relative movements and accompanying posture types that can be determined from frame to frame, such sequences can be interpreted as hand commands that are to some extent incorporated into user interfaces for computers, mobile devices and integrated systems.

The present invention provides interaction systems that use pen, touch or both (dual mode systems) suitable for teaching, collaboration and meetings. Current operating systems and graphical user interfaces are prepared for dual-mode, multi-touch and multi-pen input, and can distinguish between touch, pen and mouse input. By combining interaction objects and pens with optical patterns and other objects, such as fingers and the hand, image recognition and pattern matching can be used to distinguish between these input modes and provide the dual-mode information from various interaction objects to the computer as multi-touch, multi-pen and mouse information simultaneously to the computer. Several new interaction platforms also allow simple pen or finger gesture control, and/or also hand gesture-based interaction.

The invention can be used in connection with interactive tables, boards and whiteboards in classrooms, lecture halls, meeting rooms, video conference rooms and colloquium rooms. The invention can be used with short-range or long-range data projectors, or with flat screens such as LCD screens, plasma screens, OLED screens or rear projection systems without reducing image quality or wearing out the equipment. Technology based on the present invention can be easily adapted to different screens, projectors and display devices with low costs and little effort.

The present invention is ideal for short-range devices, typically mounted on the wall, since it can be integrated into or attached along the wall projector, or attached to the projector wall mount to make installation simple and robust.

The present invention can also be used in lecture halls where very long interactive whiteboards and interaction rooms are necessary to provide touch, pen and gesture control, and can also interact with pointing sticks and laser pointers and be tolerant of and adaptable to different display formats.

The present invention can also be used together with flat screen technologies to make them interactive, including position and gesture control. Since the present invention is based on the use of CMOS image sensors and image signal processing, the system can exhibit a high coordinate update rate and provide low latency that provides the best user experience.

The interaction systems according to the present invention can be very easily converted to fit into existing infrastructure to, for example, upgrade already installed pen-based interactive whiteboard models to also enable finger touch and hand gesture control, or to upgrade an existing meeting or teaching room equipped with a projector or flat screen, to become interactive with a simple installation of the input device itself.

The interaction system according to the present invention can also be used in interactive vertical and horizontal multi-user surfaces in multi-user rooms and control rooms, museums and exhibitions, in interactive guest tables in restaurants and in digital posters in public and commercial areas outdoors and indoors.

The present invention will also provide advanced multi-touch interaction for users in the educational and business markets. The present invention will be suitable for small and medium-sized display devices as well as large and wide whiteboards in schools and in lecture halls.

The present invention can also be used with or without a display device in teaching, for interactive information display and in museums and exhibitions.

Description of the drawings

The invention is described here, by way of examples only, with reference to the attached drawings, where: Figure 1 is an illustration of an example of a design according to a preferred embodiment of the present invention, where a mirror arrangement of axially displaced, mainly parabolic elements is organized in a semi-circular fashion around a short-throw projector holder; Figure 2 is a presentation of an embodiment as shown in Figure 1, in side view; Figure 1B is an illustration of an embodiment according to a preferred embodiment of the present invention, where a mirror arrangement of axially displaced, mainly parabolic elements is organized in a semi-circular shape above the flat screen; Figure 2B is an illustration of a design as shown in Figure 1B in side view; Figure 3 is an illustration of an example of a design according to a preferred embodiment of the present invention, where a mirror arrangement of axially displaced, mainly parabolic elements is organized along a straight molding just above a viewing area for a projector; Figure 4 is a presentation of an embodiment as shown in Figure 3, in side view; Figure 3B is an illustration of an example of a design according to a preferred embodiment of the present invention, where a mirror arrangement of axially displaced, mainly parabolic elements is organized along a straight strip just above a flat viewing area; Figure 4B is a presentation of a design as shown in Figure 3B in side view; Figure 5 is an illustration of an example of a design according to a preferred embodiment of the present invention, where a mirror arrangement of axially displaced, mainly parabolic elements is organized to avoid obstacles such as, for example, a stand for a short-throw projector or a mount , to place the mirror elements in areas in direct lines of sight from a camera arranged outside a viewing area; Figure 6 is a presentation of a design as shown in Figure 5, in side view; Figure 7 is an illustration of an example of a design according to a preferred embodiment of the present invention, where a mirror arrangement of axially displaced, mainly parabolic elements is organized in a semi-circular shape on a table near a projector or camera mount; Figure 8 is a presentation of a configuration as shown in Figure 7, in side view; Figure 7B is an illustration of an example of a design according to a preferred embodiment of the present invention, where a mirror arrangement of axially displaced, substantially parabolic elements is organized on a table near a camera mount and a flat screen; Figure 8B is a presentation of a design as shown in Figure 7B, in side view; Figure 9 is an illustration of an example of a design according to a preferred embodiment of the present invention, where the mirror arrangement of axially displaced, mainly parabolic elements is organized along a straight strip just above the viewing area of the projector for a rear projection system; Figure 10 is a presentation of a design as shown in Figure 9, in side view; Figure 9B is an illustration of an example of a design according to a preferred embodiment of the present invention, where a mirror arrangement of axially displaced, mainly parabolic elements is organized along a straight strip just above a display area of a transparent screen (for example, an OLED system) ; Figure 10B is a presentation of a design as shown in Figure 9B, in side view; Figure 11 is an illustration of an example of a design according to a preferred embodiment of the present invention, where a mirror arrangement of axially displaced, mainly parabolic elements is organized along a straight strip just above an upper side of a viewing area for a rear projection system mounted in a table; Figure 12 is a presentation of a design as shown in Figure 11, in side view; Figure 11B is an illustration of an example of a design according to a preferred embodiment of the present invention, where a mirror arrangement of axially displaced, mainly parabolic elements is organized along a straight strip just above a display area for a transparent screen (for example OLED) mounted on a table; Figure 12B is a representation of a design as shown in Figure 11B, in side view; Figure 13 is an illustration of an example of a design according to a preferred embodiment of the present invention, where a mirror arrangement of axially displaced, substantially parabolic elements is organized along a circular shape, for example over an upper side of a projector display area for a backlight projector system mounted in a table, or organized items in areas of direct line of sight from a camera to avoid obstructions, but outside the viewing area; Figure 14 is a representation of a design as shown in Figure 13, in side view; Figure 13B is an illustration of an example of a design according to a preferred embodiment of the present invention, where a mirror arrangement of axially displaced, mainly parabolic elements is organized along a circular shape, for example over an upper side of a transparent display screen (for example OLED) mounted in a table, or organized into elements with a direct line of sight from a camera to avoid obstructions, but outside the viewing area; Figure 14B is a presentation of a design as shown in Figure 13B, in side view; Figure 15 is an illustration of an example of a design according to a preferred embodiment of the present invention, where a mirror arrangement of axially offset, substantially parabolic elements is organized along a circular shape, for example over an upper side of a projector display area for a wall-mounted rear projection system , or organized into elements in areas with direct line of sight from a camera to avoid obstructions, but outside the viewing area; Figure 16 is a presentation of a design as shown in Figure 15, in side view; Figure 15B is an illustration of an example of a design according to a preferred embodiment of the present invention, where a mirror arrangement of axially displaced, mainly parabolic elements is organized along a circular shape, for example over an upper side of a transparent display screen (for example OLED) mounted on a wall, or organized into elements in areas with a direct line of sight from a camera to avoid obstructions, but outside the viewing area; Figure 16B is a presentation of a design as shown in Figure 15B, in side view; Figure 17 is an illustration of an example of a design according to a preferred embodiment of the present invention, where a mirror arrangement of axially displaced, mainly parabolic elements is organized along a straight strip just above the display area of a transparent screen (for example OLED) mounted in a hand-held device; Figure 18 is an illustration of typical camera images for some examples of designs according to preferred embodiments of the present invention, where the mirror arrangement of axially displaced, mainly parabolic elements is organized in several different ways; Figure 19A is an illustration of a parabola and an off-axis segment; Figures 19B to 19F are illustrations of exemplary designs of off-axis, concave and substantially parabolic elements, and also illustrations of some manufacturing limitations; Figure 20 is an illustration of examples of designs of mirror elements according to preferred embodiments of the present invention; Figure 21A is a flowchart illustrating an example of a methodology that facilitates finding the distance of the fingers to a surface, finding the three dimensional coordinates of the fingers within a volume, and a touch and hover condition for the fingers; Figure 21B is a flow chart illustrating an expedited methodology for finding a object; Figure 22 is an illustration of examples of methodologies that facilitate calibration of a camera to a display screen according to a preferred embodiment of the present invention; Figure 23 is an illustration of exemplary designs of a stylus with tracking patterns, a mirror with location control patterns, and a coordinate plane with location control patterns; Figure 24 is an illustration of example configurations for providing a controlled background for imaging and for measurement using a small strip along one or more edges of a coordinate plane; Figure 25 is an illustration of exemplary designs of a mirror arrangement of off-axis, substantially parabolic elements at a coordinate plane combined with additional curved or planar mirror elements further outside the coordinate plane, to provide spatial information observed using a camera; Figure 26 is an illustration of examples of designs of a mirror arrangement of axially displaced, mainly parabolic elements at a coordinate plane for observing an object's height in relation to a coordinate plane, combined with a separate device for illumination; Figure 27 is a schematic illustration of a system comprising a display device, a cooperating computer and a device according to the present invention; Figure 28 is an illustration of an example of a design according to a preferred embodiment of the present invention, where a direct image and a mirror image are captured using two cooperating, separate image sensors with optics to optimize each image for low manufacturing costs, miniaturization and simple setup; and Figure 29 is a schematic illustration of an example of a design of a mirror arrangement with two sections consisting of axially displaced, mainly parabolic mirror elements M1 and M2 which can be observed by a camera, and an object P which is located in an interaction volume.

Description of embodiments of the invention

The present invention relates to a device, a system and a method for a camera-based data input device for man-machine interaction. The present invention also relates to devices for implementing such systems and performing such methods.

Before explaining at least one embodiment of the invention in detail, it should be noted that the invention is not limited in its application to those details of construction and arrangements of components set forth in the following description or illustrated in the drawings. The invention may be implemented by means of other embodiments or practiced or carried out in different ways. It should also be noted that the phraseology and terminology used herein is for the purpose of description and should not be considered limiting.

The principles and operation of the device, system and method for interactive input according to the present invention can be better understood with reference to the drawings and their accompanying description.

First, the principle of an interaction device and an interaction system will be described. Next, detailed description of some preferred embodiments will be described together with their detailed principles of operation.

The principle behind the interaction device and the interaction system is described with reference to an example of a design as illustrated in Figure 1 and Figure 2 which schematically outlines a hardware configuration of a preferred embodiment of the present invention as shown in perspective and in a side view. The equipment components in this embodiment are a short-range data projector 3 placed together with a camera 5 and a lighting source 6 on a wall mount 4.

The appearance and the practical implementation of the wall mount 4 can vary considerably, but a main purpose of this is to place one or more of the short-range projector 3, the camera 5 and the illumination source 6 at the correct distance to a screen and for mounting on the wall 11, preferably above a projected image 12. The projected image 12 also represents the coordinate plane 12 and is preferably projected onto a smooth and white surface suitable for projection, pen operation and touch, while in the case of using a planar display device, the interaction surface 12 is the display device itself, possibly protected with a special transparent material, typically glass or plastic material for protection, so that it is robust to pen and touch operations. The computer projector 3 has a field of view 9 and can be operated to project the displayed image 12 as represented by the solid rectangle in an interaction volume 1.

An object 2, which is for example the user's finger and/or hand, can interact with a computer or the like within the interaction volume 1 which is limited by a particular height above the coordinate plane. It includes a mirror arrangement 7 with at least one axis-shifted, mainly parabolic element on the outside of the interaction volume 1, with its axis parallel to the coordinate plane and its parabolic focal point at the camera's entrance pupil to provide a constant magnification of the volume's height dimension along its axis.

The camera 5 has a field of view 8 which includes the interaction volume 1 and the mirror arrangement 7, so that the object's coordinates and the object's hover height can be calculated and/or its hover state and/or touch state and/or the position characteristics can be derived based on a single image processed by the calculation unit, and /or the object's movement and/or the object's gestures can be calculated in more detail based on a sequence of images processed by the calculation unit, where the calculation unit is typically, but not necessarily, integrated in the camera 5. The camera 5 can have optical filters to selectively block light of different wavelengths -areas, for example to reduce the impact of daylight and light from the display device. The camera 5 can also be equipped with a bifocal lens to enlarge the mirror arrangement 7 at the expense of its surroundings, thereby increasing the resolution of the image of the mirror arrangement 7 in the sensor pixel group of the camera 5.

The computing device has communication means, for example a microcontroller, for transferring the coordinates and the other interaction data to a computer using, for example, a standard serial bus and circuitry (such as USB) or using wireless communication protocols and devices.

The lighting device 6 can be directional and switchable, thereby illuminating the object 2 either directly or through the mirror arrangement 7, so that the most correct lighting can be selected for the lateral position determination and the detection of the hovering height, respectively. For lateral positioning of the object, illumination through the mirror arrangement 7 may be preferable due to the formation of a field of light rays of substantially constant height parallel to the plane which will illuminate the object from the side as it enters the interaction volume 1, thus also providing a certain contouring of the object 2 when observed directly from the camera 5. For determination of hovering height, however, direct illumination may be more attractive (than illumination through the mirror arrangement 7) to thereby separate the optical paths of the illumination device 6 and the camera 5, in order to maximize signal/ the noise ratio, and further provide some contouring of the object 2 when viewed through the mirror arrangement 7. In some embodiments, the lateral illumination can also be done by means of a substantially similar mirror arrangement, which is separate from the mirror arrangement 7 arranged to be optimized for observation to get the best signal/noise -the ratio for the combination of lateral illumination and lateral observation.

In all the embodiments and the preferred embodiments according to the present invention, it may further include at least one external screen or

frame, which is omitted here to clarify the figures, but which can enclose one or more of the equipment components: the projector 3, the camera 5 (including the calculation units and the communication means), the light source 6, the wall mount 4, the mirror arrangement 7 and the display device and the coordinate plane 12. Purpose with the outer screen or frame can for example be to make the interaction system robust, maintenance-free, dust-proof, user-friendly, safe, easier to manufacture, easier to assemble and to present the system with a professional appearance according to some given principles and design elements.

Reference is made further to figure 1 and figure 2, where the mirror arrangement 7 of axis-displaced, mainly parabolic elements in this design example is arranged in a mainly semi-circular curvature above the coordinate plane and the display device 12, preferably either mounted on the wall 11, on the projector mount 4 or on the the surface extending along the coordinate plane and the display device 12.1 this preferred embodiment, the mirror arrangement 7 can be an integral part of the wall mount or an integral part of the entire interactive whiteboard. The mirror arrangement 7 may also be included in a conversion kit for upgrading an existing whiteboard or short throw projector installation to become touch sensitive.

Referring to Figure 1B and Figure 2B, the embodiment illustrated here is similar to that described above in connection with Figure 1 and Figure 2, except that the projector 3 and the projector display surface 12 are replaced by a flat screen (LCD, plasma , OLED, backlight projection, etc.) for the display device 12.

Reference is still made to Figure 1B and Figure 2B, where a self-contained embodiment without any display device 12 can be used to capture, for example, precise strokes from a chalk and a sponge and finger touches on a traditional blackboard, while the captured results are stored in a computer and the input or some interpretation of the input is displayed by means of the conventional computer screen or by means of an attached display device or a projector for reference to the user and/or spectators.

Reference is made to Figure 3 and Figure 4, where the mirror arrangement 7 comprises axis-shifted, mainly parabolic elements placed along a straight line outside an edge, preferably an upper edge of the display device and the coordinate plane 12. The same properties and functions as described in connection with Figure 1 and Figure 2 still apply except for a difference regarding the physical appearance of the mirror arrangement 7.

Reference is made to figure 3B and figure 4B, where the embodiment is the same as that described above in connection with figure 3 and figure 4, except that the projector 3 and the projector display surface 12 are replaced by a flat screen (LCD, plasma, OLED, backlight projection etc.) the viewing surface 12.

Reference is made to figure 5 and figure 6, where the mirror arrangement 7 comprises axially displaced, mainly parabolic elements placed in areas for direct insight from the camera 5 in order to avoid obstacles caused, for example, by the projector's 3 chassis or wall mount 4, while they are outside the viewing surface and the coordinate plane 12 The same characteristics and functions described in connection with Figure 1 and Figure 2 still apply, except for the physical appearance of the mirror arrangement 7.1 In some configurations, the projector 3 and the projector display surface 12 are replaced by a flat screen (LCD, plasma, OLED, backlight projection, etc.) instead of the viewing surface 12.

Reference is made to Figure 7 and Figure 8, where the same features and functions described in connection with Figure 1 and Figure 2 apply, except that the system is not mounted for vertical use on a wall, but instead is mounted for horizontal use on a table surface 12.

Reference is made to figure 7B and figure 8B, where the embodiment is the same as that described above in connection with figure 7 and figure 8, except that the projector 3 and the projector display surface 12 are replaced by a flat screen (LCD, plasma, OLED, backlight projection, etc.) as display surface 12.

Reference is made to Figure 9 and Figure 10, where the same properties and functions described in connection with Figure 3 and Figure 4 apply, except that the system is now adapted to a semi-transparent backlight projection screen 12 so that the camera 5, the light source 6, the projector 3 and wall mount 4 are behind the wall 11, while the mirror arrangement 7 with axially displaced, mainly parabolic elements along a straight strip, is mounted above the projection screen 12 on the wall to observe the interaction volume 1 at a certain given height above the viewing surface and the coordinate plane 12.

Referring to Fig. 9B and Fig. 10B, the embodiment is similar to that described above in connection with Fig. 9 and Fig. 10, except that the projector 3 and the display surface 12 of the projector are replaced by a semi-transparent flat screen (OLED, etc.) as the display surface 12.

Reference is made to Figure 11 and Figure 12, where the same features and functions described in connection with Figure 9 and Figure 10 still apply, except that the system is not mounted for vertical use on a wall, but is instead mounted for horizontal use on a table surface 12.

Reference is made to Figure 11B and Figure 12B, where the embodiment is the same as that described above in connection with Figure 11 and Figure 12, except that the projector 3 and the projector display surface 12 are replaced by a semi-transparent flat screen (OLED, etc.) as the display surface 12 .

Reference is made to Figure 13 and Figure 14, where the same properties and functions described for Figure 11 and Figure 12 apply, except that the mirror arrangement 7 of axially displaced, mainly parabolic elements is organized along a circular shape, for example over an upper side of the projector viewing area for a backlit projection system mounted in a table, or organized into elements in areas for the direct line of sight from the camera to avoid obstructions, but outside the viewing area.

Reference is made to Figure 13B and Figure 14B, where the design is similar to that described above in connection with Figure 13 and Figure 14, except that the projector 3 and the projector display surface 12 are replaced by a semi-transparent flat screen (OLED, etc.) as the display surface 12.

Reference is made to figure 15 and figure 16, where the same characteristics and functions described in connection with figure 9 and figure 10 apply, except that the mirror arrangement 7 of axially displaced, parabolic elements is organized along a circular shape, for example over the upper side of the projector viewing area 12, or organized into elements in areas with direct line of sight from the camera 5 to avoid obstructions, but outside the viewing area 12.

Referring to Figure 15B and Figure 16B, the configuration is similar to that described above in connection with Figure 15 and Figure 16, except that the projector 3 and the projector display surface 12 are replaced by a semi-transparent flat screen (OLED, etc.) as the display surface 12.

Reference is made to Figure 17, where the same features and functions as described in connection with Figure 9B, Figure 10B, Figure 11B and Figure 12B still apply, except that the interactive system is designed to be mounted in a handheld device.

Reference is made to figure 18, where typical images of some embodiment examples according to the present invention are illustrated, where the mirror arrangement 7 of axially displaced, mainly parabolic elements organized (a) along a circular shape as in figure 1, figure 2, figure 1B, figure 2B, figure 7, Figure 8, Figure 7B, Figure 8B; (b) along a straight strip parallel to an edge of the coordinate plane 12 as in Figure 3, Figure 4, Figure 3B, Figure 4B; (c) along elements in areas with direct line of sight from the camera 5 to avoid obstacles, as in Figure 5 and Figure 6; (d) along two, three or four straight strips parallel to the edges of the coordinate plane 12, which can provide multiple images of the object 2;

(e) along a straight long strip parallel to the upper edge of a very wide coordinate plane 12 covered by viewing points of several cameras 5; (f) along one or more elements in areas of the direct line of sight from the cameras 5 to avoid obstacles, and which can provide several different images of the object 2. This design can also be used in connection with interactive information signs and in interactive posters in fairs and museums where several interactive areas or islands can be established between areas with, for example, three-dimensional structures with information content that the user can interact with. Reference is made to Figure 19, where a parabola with focal point 2' is described using the equation and an example of an off-axis, substantially parabolic element (above the shaded area and inside the dashed oval) is shown. Now, examples of numerical values will be given for a semi-circular mirror arrangement 7 of parabolic elements for a camera 5 with an entrance pupil located x=510 mm away from the viewing surface 12, and with an outer radius of R=150 mm, and a height H=50 mm (which means that an interaction volume 1 with a height of 50 mm can be observed through the mirror arrangement 7). The focal point is

The distance R-r from the outer radius as a function of the actual height h to the surface of the parabolic element, where R is the outer radius and small r is the actual radius, can be found for some height values h as follows:

Reference is made to figure 19B, where an example of a mirror arrangement 7 is shown in connection with the numerical example above, where the axially displaced, concave parabolic elements are arranged in a sector of 176° of a circle with an outer radius of 150 mm. The mirror arrangement 7 has a height of 0-50 mm, while the total height of the unit is 60 mm. The part can be injection molded in ABS plastic and metal coated with aluminum and protected by a thin polymer layer to avoid deterioration due to oxidation. Alternatively, metal-coated plastic sheets can be glued to the part, but then the correct double-curved surface is not easy to form.

Reference is made to Figure 19C, which shows the shape of a sheet of metal-coated plastic material for the example of mirror arrangement 7 as indicated in Figure 19B and related to the above numerical example.

Reference is made to figure 19D, where a perspective drawing of the example of mirror arrangement 7 as described in figures 19A, 19B and 19C is shown. The mirror arrangement 7 is adapted for placement directly on the surface extending on the coordinate plane 12 or the same level mounted on the wall 11 or wall bracket 4.

Reference is made to figure 19E, where an example of a mirror arrangement 7 can be designed, which due to certain manufacturing limitations in a given case only allows the mirror surface to be single curved. Figure 19E is an illustration of the different form of the ideal axis-shifted parabolic function and this linearized axis-shifted parabolic function. The slope of the single curved surface is adjusted to be almost correct at height=0, which means reading the "final touch" at h=0 will be quite correct. For the mirror arrangement 7 with the ideal parabolic function, the reading through the mirror of the object's height above the coordinate plane 11 will directly be a linear function and independent of the relevant (X,Y) position in the interaction volume 1, while for the mirror arrangement 7 using such a prepared, non-ideal parabolic function, the reading of the object's height will need to be corrected using an (X,Y) position-dependent error expression, for example implemented using a lookup table.

Referring to Figure 19F, an example of a mirror arrangement 7 is designed which, due to certain manufacturing limitations, for example, is limited to having two single curved surfaces, namely the two linear sections to approximate the off-axis concave ideal parabolic shape . The figure illustrates the difference in shape between the ideal off-axis, parabolic function and the off-axis, essentially parabolic function that has two linearized sections. These shape artifacts will distort the image of the object, since the deflection angles are not correct. In general, it is possible, for example due to manufacturing limitations, to utilize various linearized, segmented or other approximate functions to approximate the off-axis, concave, parabolic function as shown, for example, in Figure 19A, and such a resulting off-axis, concave, substantially parabolic element can provide sufficient image quality to observe the object and determine the object's hovering height with sufficient accuracy according to given system requirements, well adapted to the sensor's final image resolution and the camera's given lens quality.

Reference is made to figure 20, where examples of designs of the mirror elements according to a preferred embodiment of the present invention comprise: (a) a mosaic of small axially displaced, mainly parabolic mirror elements; (b) mirror-like films of metal-coated plastic material bonded to a substrate; (c) mirrors using total internal reflection in glass or plastic material; (d) mirrors using total internal reflection in glass or plastic material and using metallization for protection and extension of the mirror function for angles smaller than the critical angle for total internal reflection; (e) mirror using a plane mirror and one or more Fresnel lenses to provide the necessary curvature for the off-axis, essentially parabolic function when the camera is in front of the screen (front projection), (f) mirror using a plane mirror and one or more Fresnel lenses to provide the necessary curvature for the off-axis, essentially parabolic function when the camera is behind the screen (rear projection or "transparent" transparent flat panel, such as OLED); (g) mirror using Fresnel-like segments for the off-axis, essentially parabolic function equivalent to (e) and (f) .

Reference is made to Figure 21 A, where a flowchart provides an illustration of an example of a method that facilitates finding finger distance to a surface, finding finger three-dimensional coordinates within a volume, and touch and hover condition. The off-axis, mainly parabolic mirror elements represent an alternative viewpoint for observing the objects, and the mirror elements explicitly represent the hover level or height or orthogonal distance Z of the object above the interaction surface inside the interaction volume. Simple image acquisition and feature extraction as outlined in Figure 21A can find positions for candidate objects within the two regions of interest in the set of camera images, namely within the direct, or synonymously, the forward viewpoint, and the mirror viewpoint. For each distinct image, a solid angle intercepted by the candidate object at the camera's entrance pupil can be found. In the mirror view, the height Z above the interaction surface (12) is found explicitly, and the correspondence problem regarding the adaptation of one or more points in the three-dimensional space by means of two observations and image processing of two different two-dimensional images will be significantly simplified.

Figure 21B is a flowchart illustrating a faster method of finding an object. In this example, the mirror arrangement is an off-axis, substantially parabolic mirror section as illustrated, for example, in Figures 19B to 19D, and with a typical image, Figure 18A, where an object is seen both through the mirror and directly.

The height Z and the angle AZIMUTH of an object (2) can be observed by the camera through the mirror which represents a rectilinear path in the coordinate system of the interaction volume (1). This straight line in the three-dimensional interaction volume (1) represents all the possible (X,Y) positions that the object (2) can have for the given Z and AZIMUTH value. This three-dimensional path is, by means of coordinate transformation for the lens, assigned to a two-dimensional path in the camera's pixel group which can for example be found using a look-up table, and this path can be traversed by starting from the end closest to the mirror, and with a certain path width given in the number of pixels , an edge detector algorithm can find a candidate object. Then subpixel edge detection or template matching can be performed to find the pixel position (X,Y) with higher accuracy, and then transformed using an inverse coordinate transformation using, for example, a lookup table, where the candidate object's coordinates with high accuracy (X,Y) in surface volume coordinates are calculated. After this search algorithm, finally X,Y,Z and position information can be reported as described.

Compared to a complete search algorithm in the two-dimensional pixel group n with an edge detector algorithm, whose computational complexity is proportional to the size of the group of interest covering the interaction volume (1), the described algorithm is much less complex and is mainly proportional to the length of the diagonal of the group as that the acceleration factor can be significant, in the range from 100x-1000x, depending on the resolution of the sensor and the area of interest.

Reference is made to Figure 22, where examples are given of methods that facilitate calibration of the camera to the display screen according to a preferred embodiment of the present invention, where (a) is a standard manual calibration solution where crosses are displayed on the display screen and an operator uses a pen or finger to touch each cross in a given order; (b) is an automatic calibration solution that uses patterns such as in the inventions WO2001NO00369 / US7083100B2 and/or WO2006135241A1 / US2009040195A1 to identify the various calibration points, in which these inventions are hereby incorporated by reference; (c) is a semi-automatic calibration solution that uses patterns such as in (b) first to identify calibration points, then present a set of white circular disks on a black background in given positions where the operator in a given sequence places a semi-transparent cylinder with internal opaque or reflective material so that touch detection limits can be set or regulated.

Reference is made to figure 23, examples of designs of (a) a pen with tracing patterns 13; (b) a mirror with position control patterns 13; (c) a coordinate plane with location control patterns 13; used in conjunction with the present invention, is shown. The patterns can, for example, be patterns used to identify and track objects such as in WO2001NO00369 / US7083100B2 and/or WO2006135241A1 / US2009040195A1 which are hereby incorporated by reference. Reference is made further to Figure 23, where the pen input by using such patterns and pattern recognition can be distinguished from other interaction devices for input such as a human finger, so that systems for dual mode input can be easily implemented using the present invention, and the other referenced inventions. Reference is also made to figure 23, where the interaction surface and the mirror can also be equipped with such patterns, so that automatic control, calibration and self-adjusting setup can be realized by using the present invention together with the other referenced inventions.

Reference is made to Figure 24, where examples of configurations for providing a regulated background for the imaging and measurements using a small injection molded article or strip 15 along one or more edges of the coordinate plane, which are typically white, black or have a reflective optical property 14 in the relevant near-infrared wavelength range. In this example, the list also serves as a pen shelf 15 below the coordinate plane.

Reference is made to figure 25, where examples of configurations of a mirror arrangement with axially shifted, mainly parabolic elements at the coordinate plane are shown, adapted to detect the height of the object above the plane, while further curved or planar mirror elements 16 further outside the axially shifted, mainly parabolic mirror elements are arranged for to provide spatial information about the scene when these mirrors are observed from the camera's point of view. This embodiment can improve the ability to follow and determine the position and gestures of objects 2 also outside the interaction volume 1 by observing the objects 2 in the mirrors 16. In a more advanced human-machine interaction scenario, the user gestures and behavior can be analyzed by observing the live image and the image in the mirror 16 to predict new interaction events. The three-dimensional position and position of the object 2 can also be estimated.

Reference is made to figure 26, where examples of designs of a mirror arrangement with axis-shifted, mainly parabolic elements at the coordinate plane are shown for observation of the object's height in relation to the coordinate plane, combined with other lighting devices 17 to provide illumination, so that the mirror arrangement 7 itself for observation of the object's height above the coordinate plane is less exposed to the direct illumination, thereby reducing unwanted reflections of the optical interfaces and thereby increasing the signal/noise ratio in the measurements.

Reference is made to figure 27, where a system is shown which comprises a display surface 12, a cooperating computer 18 and the device 19 according to the present invention, and communication means 20 between the cooperating computer and the display device 12 and the communication means 21 between the cooperating computer and the present device 19 according to present invention. The communication means 20 are optionally implemented as a wireless data connection and/or a direct cable-connected connection and/or an optically modulated connection.

Reference is made to figure 28, which shows an example of a design according to a preferred embodiment of the present invention, where the direct image and mirror image are captured by means of two interacting, separate image sensors, 23 and 24 respectively, with separate optics to optimize each image, for low cost, miniaturization and easy set-up, and connected through, for example, a high-speed serial connection 22. The dashed line 10 indicates that one or more of the various components may be enclosed by a housing 10. A separate lighting unit 17 as shown in FIG. 26, may also be contained within this housing 10. However, the components may also be separate and modular for retrofitting into an existing projector installation to make it interactive or, for example, to upgrade a pen-based interactive whiteboard to become touch-sensitive. Lens optics that are best suited for the two separate viewing angles are preferably used, and then the same calculations are carried out on the image pair using the calculation unit. The acceleration method described in Figure 21B for the present invention will also apply with the same acceleration potential in such dual sensor/lens configurations.

Reference is made to Figure 29, where a redundant way to find the interaction object (2) and touch and hover state in the case of blocking the direct camera image is inspired by the acceleration procedure described in Figure 21B, applied to, for example, two mirror arrangements 7: mirror M1 and mirror M2, where a distance between the mirrors M1 and M2 is a baseline L as shown. Corresponding to the procedures in figure 21B, one can find the azimuth a and height Z1 of the object P by observing the mirror M1 and the azimuth (3 and height Z2 of an object P by observing the mirror M2, and by triangulation find the object position (X,Y) on the interaction surface 12 or in the interaction volume 1.

The two mirrors M1 and M2 are placed at a distance L from each other, that is, the baseline is L. Then the distance d from the baseline with length L to the target P is equal to:

The distance can also be expressed as:

The X and Y coordinates can be easily derived using simple trigonometric calculations.

Using coordinate transformation or using a lookup table, the (x,y) position of the corresponding sensor image can be found, and a detailed image analysis can be performed locally in an area around the (x,y) position to obtain a more accurate position determination which by means of coordinate transformation or look-up table can be transformed to a correspondingly precise (X,Y) position on the interaction surface (12) or the interaction volume (1) in the coordinate system.

Modifications of embodiments of the invention described above are possible without deviating from the scope of the present invention as defined in the appended patent claims. Expressions such as "comprising", "encompassing", "incorporating", "consisting of", "having", "is" used to describe the present invention are intended to be understood in a non-exclusive manner, namely to make it is possible for articles, components or elements that are not explicitly described to also be present. Reference to the singular shall also be understood to refer to the plural. Figures enclosed in parentheses in the appended claims are intended to assist in understanding the claims and shall not in any way be deemed to limit the scope of these patent claims.

Claims (38)

1. Device for determining a position or a position or both, of at least one object (2), where the object (2) is wholly or partially located within an interaction volume (1) delimited by an interaction surface (12) and by a certain height area in a height dimension above the interaction surface (12), comprising - a camera (5) - a mirror arrangement (7) comprising one or more mirror sections; - a calculation unit for calculating position and position or both, for at least one object (2) based on information from, among other things, the camera (5); where the camera (5) is arranged to include both the volume (1) and the mirror arrangement (7) within the camera's (5) field of view; - the mirror arrangement (7), where the one or more mirror sections comprise at least one axially shifted, concave, mainly parabolic optical mirror element at the plane of the interaction surface (12), and where each axially shifted, mainly parabolic optical mirror element is arranged with its focal point at the entrance pupil of the camera and its axis parallel to the surface (12) so that an image of the volume (1) is produced with constant magnification of the height dimension for each substantially parabolic optical mirror element along its axis; so that the object's (2) position and/or position is determined by the calculation unit based on information from a single image from the camera (5). 2. Device according to claim 1, comprising only one camera (5). 3. Device according to claim 1 or 2, where at least one second object is within the camera's (5) field of view, but not necessarily within the volume (1), where the position of the at least one other object is determined, so that the position of the second object can provide additional information. 4. A device according to any one of the preceding claims, wherein the off-axis concave substantially parabolic optical mirror element comprises a Fresnel-like mirror element to provide an off-axis concave substantially parabolic mirror function. 5. Device according to any one of claims 1 to 3, wherein the off-axis, concave, substantially parabolic optical mirror element comprises a mirror element and a lens element arranged in combination to provide the off-axis, substantially parabolic mirror function. 6. Device according to claim 5, where the mirror element is linear. 7. Device according to claim 5 or 6, where the lens element is a Fresnel lens. 8. Device according to any one of the preceding claims, where the mirror element comprises a reflective surface where reflection is provided either by means of a film of a metal-coated plastic material, injection molded parts of metal-coated plastic material, by means of internal total reflection or by means of internal total reflection combined with metal coating. 9. Device according to any one of the preceding claims, where the mirror element comprises a layer of plastic material and/or a special coating that selectively stops or allows light to pass through within given wavelength ranges, so that, for example, the mirror element will become functional in the near infrared light with reduced reflections of visible light. 10. Device according to any one of the preceding claims, where at least one mirror section is arranged to be arranged on an edge of a periphery of the interaction surface (12). 11. Device according to any one of the preceding claims, where at least one mirror element is arranged in a straight injection molded strip along an edge of the edge of the interaction surface (2). 12. Device according to any one of claims 1 to 9, where the at least one mirror element is distributed over a semi-circular shape designed to be arranged by a wall or table mount. 13. Device according to any one of the preceding claims, where the mirror arrangement (7) comprises a number of mirror sections, and the mirror sections are arranged to provide several viewing angles of the object (2). 14. Device according to claim 13, where the mirror elements are arranged in a mosaic structure in order to reduce shadow formation and emphasize assignment characteristics from mirror to pixel. 15. Device according to any one of the preceding claims, comprising a number of cameras, where the cameras are arranged to provide multiple viewing angles for the at least one object (2) and in areas with a direct line of sight from the cameras to avoid shadowing. 16. Device according to any one of the preceding claims, wherein at least one camera (5) is arranged with a bifocal lens to enlarge the image of the at least one part of the mirror arrangement (7). 17. Device according to any one of the preceding claims, wherein at least one camera comprises at least one optical filter to block or pass light at a selected wavelength so that unwanted light is stopped while light in the wavelength range of the illumination is allowed to pass. 18. Device according to any one of the preceding claims, wherein at least one camera comprises at least one selectable optical filter to selectively block or pass light at different wavelength ranges so that, for example, light of the same wavelength as the light source or visible light is blocked or let through. 19. Device according to any one of the preceding claims, comprising a lighting arrangement (6) arranged to provide illumination of at least parts of the interaction volume (1) with visible and/or near infrared light, directly and/or indirectly via the mirror arrangement (7). 20. Device according to claim 19, where the lighting arrangement (6) is regulated to switch the light on and off and/or to provide flashing within an active exposure period for the camera (5) to freeze movements of the one or more objects (2). 21. Device according to claim 19 or 20, where the lighting arrangement (6) is arranged near the entrance pupil of the camera (5), namely near the focal point of the axis-shifted, mainly parabolic elements and direct illumination through the mirror arrangement (7) so that the illumination is spread in the interaction volume (1) with rays substantially parallel to the interaction surface (12). 22. Device according to any one of the preceding claims 19 to 21, further comprising a separate, other mirror arrangement arranged to help illuminate the interaction volume (1) so that the mirror arrangement (7) for observation is less exposed to illumination thereby increasing a signal-to-noise ratio for the measurements performed using the devices. 23. Device according to any one of the preceding claims 19 to 21, further comprising a separate, other illumination arrangement (17) arranged to help illuminate the interaction volume (1), thereby increasing the signal-to-noise ratio of measurements performed when using the device. 24. Device according to any one of the preceding claims 19 to 23, where the lighting arrangement (6) is arranged to provide direct lighting and for indirect lighting through a mirror arrangement (7), and where the direct and indirect lighting is regulated separately for providing detection of the one or more objects (2). 25. Device according to any one of the preceding claims 19 to 24, where the lighting system (6) is arranged to change an appearance of the object (2), for example by projecting a color and/or flashing lighting as interaction feedback to a user from a computer. 26. Device according to any one of the preceding claims, further comprising additional curved or planar mirror elements (18) arranged to provide spatial information when observed from the camera (5). 27. Device according to any one of the preceding claims, comprising two mirror sections arranged at a distance to enable the position or position or both of an object to be found by triangulation, so that the position or position or both can also be determined in a case of blockages in the direct camera field of view of the object. 28. Device according to any one of the preceding claims, wherein lens optics are separated for direct vision and vision through the axis-shifted, mainly parabolic mirror elements by using one or more separate sensors. 29. Interaction system for providing interactive use of an object in the vicinity of a presentation surface, where the interaction system comprises a device for determining position and/or position according to one or more of the preceding claims, where the interaction system further comprises presentation devices arranged to present images at the presentation surface. 30. System according to claim 29, comprising a front projection screen (12) where the camera (5), the light source (6), the projector (3) are arranged on the same side of the screen (12) as the interaction volume (1). 31. System according to claim 29, comprising a semi-transparent rear projection screen (12) where the camera (5), the light source (6), the projector (3) are arranged on the opposite side of the screen (12) in relation to the interaction volume (1). 32. System according to claim 29, comprising a semi-transparent flat screen (12), for example OLED, where the camera (5), the light source (6) are arranged on the opposite side of the screen (12) in relation to the interaction volume (1). 33. System according to any one of claims 29 to 32, where the interaction surface (12) is arranged by a wall, a table or a hand-held device. 34. System according to any one of claims 29 to 30, where the interaction system comprises a fastening device (4) for the projector (3), and where at least one mirror arrangement (7) is arranged in connection with the fastening device (4) so that almost optimal position determination of different components in the system is simplified. 35. Method for determining a position or a position or both, for at least one object (2), where the object (2) is fully or partially located inside an interaction volume (1) bounded by an interaction surface (12) and by a certain height range in the height dimension above the interaction surface (12), comprising the following steps: - to reflect radiation from an object inside a volume close to or inside the interaction volume by using a mirror arrangement (7) comprising at least one axis-shifted, concave and substantially parabolic optical mirror element at the plane of the interaction surface (12), and where each axis-shifted substantially parabolic optical mirror element is arranged with its focal point at the camera's entrance pupil and with its axis parallel to the surface (12) so that an image of the volume (1) is produced with constant gain of the height dimension of each substantially parabolic optical mirror element along its axis; - registering reflected radiation by means of a camera (5) arranged to include both the volume (1) and the mirror arrangement (7) within the field of view of the camera (5); - to transfer information from the camera to computing means; and - to calculate position and position or both, for at least one object (2) based on information from, among other things, the camera (5). 36. Method for calibrating and adjusting the height dimension over an interaction surface for accurate touch and hover information, comprising the method for determining the position or position or both according to claim 35, further comprising: - placing a semi-transparent, three-dimensional pattern test object on the interaction surface; - to highlight the interaction surface in circular areas one by one; - observing the test object directly and seen from the side through the mirror arrangement by means of the camera; - to identify the pattern of the test object; - to calibrate and assign from the coordinate plane coordinates to interaction surface coordinates and/or calibration of the height measurement; and - to determine thresholds for touching and hovering. 37. Method for finding an object's distance to a surface, three-dimensional coordinates and touch and hover condition, where the object can be a finger, comprising the method for determining the position or position or both, according to claim 35, further comprising: - to provide a standard image acquisition and feature extraction; - to find spatial angles that a fingertip intercepts at the camera's entrance pupil in the front view and in the mirror image; - to find the distance of the finger to the surface using a direct linear model if the mirror is a parabola or otherwise an approximate parabolic model; - to find the three-dimensional coordinates of the finger based on the spatial angles and the distance to the surface; and - finding the hover/touch state of the finger by combining distance to the surface with threshold values. 38. Method for increasing the speed of a calculation and a search for tracking objects in an interaction volume comprising the method for determining the position or the position or both according to claim 35, further comprising: - performing a standard image acquisition of an image and feature extraction within sub-image which includes the mirror arrangement; - to find the distance of the object to the surface and the effective observation angle of the parabolic mirror element along the interaction volume (1); - finding a straight line in the three-dimensional interaction volume (1) representing all the possible (X,Y) positions of the object (2); - finding a corresponding two-dimensional path in the camera's pixel array, for example via a look-up table; - traversing this path with a certain path width with an edge detector and finding a candidate object in the group (X,Y) position; - to perform detailed edge detection or pattern matching to find an accurate group position; - to find a corresponding X,Y position, when Z is known, and - to report one or more of X,Y,Z and touch and hover information to a computer.