TW543323B - Multiple camera control system - Google Patents

Abstract

A multiple camera tracking system for interfacing with an application program running on a computer is provided. The tracking system includes two or more video cameras arranged to provide different viewpoints of a region of interest, and are operable to produce a series of video images. A processor is operable to receive the series of video images and detect objects appearing in the region of interest. The processor executes a process to generate a background data set from the video images, generate an image data set for each received video image, compare each image data set to the background data set to produce a difference map for each image data set, detect a relative position of an object of interest within each difference map, and produce an absolute position of the object of interest from the relative positions of the object of interest and map the absolute position to a position indicator associated with the application program.

Description

543323 A7 B7 V. Description of the invention (1) Reference to related applications This application is US Provisional Application No. 60/23 7,187 (named "Dual Camera Control System (DUAL CAMERA CON DING ROL)) · SYSTEM) ") claims its priority and is hereby incorporated by reference. TECHNICAL FIELD The present invention relates to an object tracking system, and more particularly to a camera based on an object tracking and interface control system. Background can be seen Various types of computer system interfaces and operating systems used to control computer systems. Many of these operating systems use standardized interface functions based on generally accepted graphical user interface (GUI) functions and control technologies. The results are convenient Users who are quite familiar with the platform and / or application control various computer platforms and user applications. This is because functions and control technologies from one GUI to another can be seen everywhere. A widely accepted control technology is Use your mouse or trackball-type pointing device to move the cursor over the screen object. An action, such as clicking or double clicking on the object You can execute the GUI function. However, for some unfamiliar computer mouse operators, clicking on the GUI function is a big challenge, which will hinder their interface with the computer system. In some cases, it is impossible to Situations such as using a mouse or trackball, such as in front of a department store display window on a city street, or speaking in front of a large display in front of the crowd. Summary-4- This paper standard applies to China National Standards (CNS) A4 specification (210 X 297 mm) 543323 A7 _B7 V. Description of the invention (2) Reveal a method of tracking the target in a general form. This method includes obtaining a first image and a second image of the target representing different viewpoints. Two images, and processing the first image into a first image data set, and processing the second image into a second image data set. This method further includes processing the first image data set and the second image data set to generate and The background data set related to the background, and the first difference map is determined by the difference between the first shirt image: the shell material set and the background / shell material set, and the difference between the first image data set and the background data set The difference decides to generate a second difference map. This method also includes detecting the first relative position of the target in the first difference map and the second relative position of the target in the second difference map. The first and second relative positions generate the absolute position of the target. The steps of processing the first image into a first image data set and processing the second image into a second image data set may include activities on the first and second images. The decision of the movie sample area and the first and first images contained in the range of the moving image area to capture the moving image data set. The step of capturing the moving image data set may include one or more first and second images. Cut, rotation of the first and second images, or cropping of the first and second images. In an actual implementation, the step of capturing the moving image data set may include configuring the moving image data set as an image pixel array with rows and rows. The fetching step further includes identifying the highest pixel value within the range of each image pixel array row, and generating a data set with one column, wherein the highest pixel value identified for each row is the table row. '2 The first image processing as the first image data set and the second image processing as the second image data set may also include filtering the first and second images. Sieve 543323 A7

The dagger includes 1 first and second image edge extraction. The screening may further include processing of the π silly image data set and the second image data set, highlighting the negative image set of the first image and the background data set. Yuta ^ -s. ^ Μ ^, and the second image The difference between the data set and the moonscape data set. For the first image data set and the second-biggest scene data set, the first image data set can be generated by:-!:, Or, and related to the first image data set ~ # 景 数据 集, and Generate a first set of one or more sets of background data sets related to the second image resource set. Generate brother-group- or multiple sets of background data sets may be included within the scope of the first image data set representing the background, generate a first background set representing the largest data value 'and generate a second set—or multiple sets of background data sets may include In the second image data set fan garden representing the background, the second one representing the largest data value is generated: scene =. The generation of the first and second background sets, which can further include the maximum value of the data representing the scene n, is increased by a predetermined value to a value within the range of the first and second background sets. Generating the first set of one or more background data sets may include generating the first set of background data, which is a minimum value in the range of the _ image data set representing the background, and generating a second set of one or more background data sets. It may include generating a second background set, which is a minimum value within the range of the second image data set representing the background. The generation may further include the minimum values of the first and second background sets, which represent the minimum value of the background data, and reduce the values within the range of the first and second background sets by a predetermined value. Generating the first set of background data sets may include generating sampled first image data. -6-This paper size applies the Chinese National Standard (CNS) Α4 specification (210X 297 public love)

Binding

543323 A7

$ To generate a second set of background data sets.娣 i, person _, worker, bao, bao, yi, xi, yi, qi, qi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, xi, yi, ren, yi only, etc. automatically occur during the predetermined period, and each of them will be taken; ^ γ # background-independent data. ^ Sampling may include information: one or more sets of background data sets may be included in each background; multiple sampling of one image data set within the Wai Wai, and the second one or more sets of background data sets may include Multisampling of the second image data set within the range of each background data set.枓 Set Qianwei ... Each first background data set of the house can include a single value selected from the multiple sampling to represent the background of each unit within the range of the image data set, and each second background data set can include the hometown weight The sampling towel selects a single value that represents the background of each unit within the range of the data set. Μ In other practical implementations, the generation may include a comparison of the second set of the background image set of the first image data set and a comparison of the second set of the second image data set and the background data set. In other practical implementations, generating the first difference map may further include listing each unit in the first image data set as one of two states, and generating the second difference map may further include each unit in the second image data set. The table is one of two states, two of which are table values that are compatible with the background. "In other practical implementations, detection may include identification of clusters in the first and second difference maps, where each cluster has a unit whose state is within the scope of its related difference map, indicating a unit that is incompatible with the background Identifying clusters can also include reducing the difference map to one row, which is calculated by calculating cells in the range of rows that are incompatible with the background. Identifying clusters can also include identifying rows within the cluster and reducing adjacent rows 543323 A7 ________B7 V. Description of the invention (5) " ~ " '~ --- It is classified in the cluster. The identification line is in the cluster range. The flesh is tenfold—the daring technique can also include the identification of the middle line. May include identifying cluster-related locations. Identifying cluster-related locations may include calculating a weighted average of the units in the cluster " π detection may also include classifying the cluster as a target. The classifying cluster may also include calculating within the cluster Clusters, and only when the count exceeds a predetermined threshold, clusters are classified as targets. Clustering clusters can also include cells that are calculated in the cluster, and calculations that are classified as a difference from the difference graph. The number of incompatible unit cities', and only when the ratio of the number of units in the cluster to the total number of units exceeds a predetermined threshold, the cluster is initially classified as the target. The detection step may further include identifying the secondary cluster in the cluster. It refers to the indicator end of the target and the identification of the target's position. In the above-mentioned actual implementation, the target can be the user's hand, and this method can include using the absolute position of the target to control the application. The above actual Implementation may further include obtaining a third image and a fourth image representing targets of different viewpoints, and processing the third image into a third image data set, and processing the fourth image into a fourth image data set; The processing of the data set and the fourth image data set generates a background data set related to the background. This method may also include generating a third difference map from the difference between the third image data set and the background data ^, and the fourth image The difference between the data set and the scene data set determines the fourth difference map; the third relative position of the target in the third difference map, and the target in the fourth difference map The relative positions of the fourth detection may be generated absolute position of the object from the first, second, third, fourth and relative position of the object. 543323 A7 B7

V. Description of the invention (6 The target can be the user's hand

This method includes obtaining from at least two viewpoints and generating image data of each acquired image. In some such actual implementations, the target may be combined to control the application by using the absolute position of the target. In another aspect, control the method of tracking the target. Image; processing the acquired image, using interaction. In addition, sets; and comparing each image data set with one or more sets of background data sets to generate a difference map for each acquired image. This method also includes detecting the relative position of the target in each difference map; generating the absolute position of the target from the relative position of the target; and using the absolute position to allow the user to communicate with the computer ^

Use the relevant screen coordinates and use the mapping position as an interface with computer applications. This method can also include organizing gestures related to the target by analyzing changes in the absolute position of the target; and merging absolute positions and gestures into an interface with a computer application. In another aspect, a connection with an application program running on a computer is revealed, and the movie camera tracking system is revisited. This multi-camera tracking system includes two or more video cameras that are configured to provide a variety of viewpoints as desired and can be operated to produce a series of video images. The processor is operable to receive the series of video images and detect objects appearing in the target area. The processor can perform the following processing: generate a background data set from the video image; generate an image data set for each received video image and compare each image data set with the background data set 'to generate a difference map for each image data set; The relative position of the target in the difference map; and the target is generated from the relative position of the target. The paper size is applicable to the Chinese National Standard (CNS) A4 specification (210 X 297 mm) 543323 ΑΊ B7 5. Description of the invention (of & right Position 'and map the absolute position to the position indicator connected to the application. In the actual implementation, the' target object can be a human hand. In addition, the target area can be displayed in front of the photographic display connected to the computer. This processor can operate the target The position of the object is mapped to the position indicator, so that the position indicator position on the photographic display is aligned with the target object. The target area can be set at any distance in front of the photographic display connected to the computer, and the processor can operate the target object. The absolute position is mapped to the position index, so that the position index position on the photographic display is aligned with the position pointed by the target. Or the target area Set at any distance in front of the photographic display connected to the computer, and the processor can operate to map the absolute position of the target to the position index, so that the movement scale of the mounted object is consistent with the larger displacement of the position index on the photographic display The processor can be configured to resemble the function of a computer mouse. This can include processing = a gesture that can be estimated from the movement of the target object and resemble a 2 mouse button of a computer mouse. Right-tolerable target within a predetermined period The position of the object can be triggered by a selection action in the application. 2 pairs of line fields-the position of the target object that can be tolerated within the predetermined period, the processor can be equipped and the adult computer mouse control buttons are similar. For _ financial period θ, during interaction = display The location of the tolerable target within the boundary of the area can be triggered by a selection action in the application. 4 = The location within the boundary of the interactive display area within a predetermined period 2 The location can be tolerated. The processor can be configured to interact with the computer mouse The control button phase is in the above aspect. The 'background data set can be packed with at least part of the static structure-10-

543323 A7 --B7 V. Description of the invention ~-= Data points shown. In this practical implementation, the at least part of the static structure may include a patterned surface visible by a camera. The static structure may be a window frame _ °. Alternatively, the static structure may include a rate of light. In another example, there are 7 ^ one and the application running on the computer have, ° evening regain 1 camera tracking system. This system contains two or more video cameras that are configured to provide as many viewpoints as desired and operate a series of video images. The processor is operable to receive the series of video images and detect objects appearing in the target area. The processor can perform the following steps: = Generate background data set from video image; Generate image data set for each received video image; Compare each image data set with background data set to generate a difference map for each shirt image data set; Detect The relative position of the target in each difference map—and, the absolute position of the target is generated from the relative position of the target; the sub-area within the target frame is identified; the sub-area occupied by the target is identified; When the secondary zone is activated, an action related to the identified secondary zone is initiated; and the action is interacted with the application. In the above practical implementation, the target may be a human hand. In addition, actions related to the identified sub-regions can be performed similarly to the activation of keyboard keys associated with an application. In the related actual implementation, for a predetermined period, the position of the tolerable target in any sub-region can be triggered by an action. One or more actual implementation details are shown in the accompanying drawings and the following description. From the description, the drawings and the scope of patent application, other features and advantages will be clearly understood. Schematic description Figure 1 shows the typical implementation of the hardware components of the multiple camera system and its code. -11-This paper size applies to China National Standard (CNS) Α4 specification (21〇 X 297 public love) 543323

Type actual layout. The camera system shown in Fig. 2A is shown in Fig. 2A. Related to the typical geometry of various image intervals Typical images captured. The system is connected to a multiple camera system. Figure 2B shows the processing flowchart shown in Figure 丨 Camera-Figure 3, which is typically executed in a microcomputer program. (Tu, Tu, Ding / Ding Ding)] is a detailed flow chart of the part of the processing, especially the processing of the position of the image signal captured by the object VII, 彳 and its secluded camera. 5A is a sample image data generated from the processing shown in FIG. 4 and obtained by a camera, such as a grayscale dot map image. A π shown in FIG. 5B is a sample image data generated from the processing shown in part of FIG. 4, such as a grayscale point image. FIG. 5C does not refer to the sampled image data generated from the processing shown in FIG. 4, such as a grayscale point image. FIG. 5D does not refer to the sampled image data generated from the processing shown in FIG. 4, such as a grayscale point image. Figure 5E refers to the sample data generated from the processing shown in Figure 4 in part, showing a grayscale point image; identifying pixels that are traced in the sample and belong to the object. Figure 6 is a detailed flowchart of the processing shown in Figure 4, especially the processing included in the object given by the pixmap. The pixmap is a pixel identified by the tracked object, such as that shown in Figure 5E. Show information. The sample data shown in Figure 7A is shown in Figure 5E, which is shown in a binary dot image image. The identification processing of the data sample shown in Figure 6 has been selected in this sample. CNS) Α4 specification (21〇χ 297 mm) 543323 A7 _ B7 V. Description of the invention (10) Object related. Fig. 7B shows the sampling data of Fig. 5E, which is shown in the form of a bar, and the identification processing of the data sampling shown in Fig. 6 has selected relevant objects from the identified special points in this figure. The difference set of sampling data shown in Figure 7C is shown in the binary bitmap image, and the identification processing of the data sampling shown in Figure 6 has been selected in this sampling to be related to objects and key parts of objects. . Fig. 8 is a detailed flowchart of the processing shown in Fig. 4, especially the process of generating and maintaining the description of the background area captured by the object. The geometry shown in Figure 9A is based on Equation 3, that is, the angle defined by some objects in the field of view of the camera, which is given by the position of the image plane on the sensitive side of the object. The geometry shown in Figure 9B is based on equations 4, 5 and 6, which is the relationship between the camera position and the tracked object. Fig. 10 is an explanatory diagram of Equation 8; that is, the suppression amount can be applied to the coordinates given by the object position change, and the position can be fine-tuned. Figure 11A is an example of an application controlled by the system, where the target controls the screen pointer in a two-dimensional space. Figure 11B shows the response between the actual coordinates and the screen coordinates, which is done using the application in Figure 11A. Figures 12A and 12B are examples of applications controlled by a multi-camera control system, where the target control screen indicators are in a three-dimensional virtual reality environment. Figure 13A shows the use of gesture detection to divide the target area into detection surfaces to identify gestures related to the desired activation. -13- This paper size applies the Chinese National Standard (CNS) A4 specification (210 X 297 public directors) 543323 A7 B7 V. Description of the invention (11) Figure 13B uses the gesture detection method to divide the target area into detection Box to identify gestures related to the selected cursor direction. Figure 13C shows another method that uses gesture detection to divide the target area into detection boxes to identify gestures related to the selected cursor direction. FIG. 13D is a detailed description of the adjacent division relationship of FIG. 13C. Similar symbols in the drawings refer to similar units. Detailed Description FIG. 1 shows a multiple camera movement tracking and control system 100 connected to an image viewing system. In this implementation, the two cameras 101 and 102 scan the target area 103. A controlled or known background 104 surrounds the target area 103. When the target 105 enters the target area 103, it is tracked by the system. The target 105 can be any non-specific object inserted into the target area 103, which is generally a hand or finger of a system user. The target 105 can also be a selection device, such as an index. The series of video images obtained from the cameras 101 and 102 are transmitted by the computing device or the image processor 106. In this implementation, the computing device is a general-purpose computer that can execute additional software to provide feedback to users on the photographic display 107. FIG. 2A illustrates a typical implementation of the multiple camera control system 100. The two cameras 101 and 102 are located outside the target area. The camera can be turned so that its field of view (205 for camera 101 and 206 for camera 102) includes the target area 103 at the overlap position. This turn causes the camera 1 (H, 102 to rotate on a nearly flat axis. In this example, the ceiling, or window, or side wall, provides a controlled background 104 with distinct edges. For the camera 丨 〇1 The captured pair is -14- & Zhang scale applicable to China National Standard (CNS) A4 specifications (21〇 ^^ ----- 543323

AT ——____ B7 ^ Explanation of the invention (12) '~ The visual field of the application is shown in Figure 2B. Although the field of view captured by the camera 102 is not shown, it should be understood that it is an image of the field of view captured by the camera 101. Controlled background 205 does not completely cover all fields of view of the camera. For each of the cameras, a moving image area 208 completely contained in the controlled background 104 is visible, and the entire target area 103 is also included. The background 104 is controlled so that the background characteristics can be modeled, and the target 105, regardless of the part or the whole, is different from the background 104. When the target 105 appears in the target area, the target 105 will block part of the controlled background 104 in the moving image area 208 of the camera 101, 102, regardless of the part or the whole, captured == The image will have the selected characteristics and be compatible with the controlled background mode. In summary, if the position of the target 105 is found in the moving image area 208 of the two cameras, it can be identified and calculated. Using the position of the camera ιιι, 〇2 and > material, and the position of the camera related to the target area 103, the target 105 in the target area 103 can be calculated. This processing is performed by the image processor 106 (Figure 丨), which can be executed by software processing or hardware processing, as shown in FIG. 3. Camera image system 7 Cameras 101 and 102 are transmitted at the same time, and are captured by the image receiving modules 304 and 305 (respectively), and transmitted to the image buffers 306 and 307 (respectively) in the image processor 106. . Scenery > The image detection modules 308 and 309 independently detect the target 105 in each image and determine its relative position with respect to the camera horizon. The relative position information 310 and 311 obtained from the two camera sights are combined by the combination module 312 and regarded as being to be fine-tuned by the position fine-tuning module 3 1 3, and the target 1 in the target area 103 is determined in block 3 14 〇5 the overall appearance and location. If necessary, a specific gesture performed by the user can be detected in the gesture detection module 315. -15- This paper size applies to China National Standard (CNS) A4 (210 X 297 mm) 543323 A7

In another process or application 316, the gesture detection processing result will be described in detail. To the same image processor 106 or another processing device. Will be detailed below

Hold

• 4 The image detection modules 308 and 309 are exactly the same in the execution process. The implementation of a shadow detection module 308, 309 is shown in FIG. In block 402, the image processor 106 is used to capture the captured Z image data stored in the image buffer 3 06 or 3 07. This image data corresponds to the moving image area 208 (Figure 2β) = The image can be screened by the screening process 403 to highlight or capture the image state or characteristics of the background 104 which is different from the target 105, otherwise it will not change with time in the background 104. In some implementations, the data representing the moving image area can also be reduced by the scale module 404, thereby reducing the amount of calculation to the amount required in subsequent processing steps. Using the obtained data, the background 104 is modeled in block 405 by one or more background mode processing conditions to generate one or more descriptions represented by the background mode data 406 of the controlled background 104. Therefore, the background 104 is patterned into a desired image appearance or characteristic item. The background mode 406 is passed to a set of standards in process 407. In comparison processing 408, the filtered (from processing 403) and / or reduced (from module 404) image data is compared with these standards (from processing 407), and the current data is compared with the background mode data 406 Incompatibilities, that is, inconsistencies with the standard, are stored in the image or difference map 409. In the detection module 410, the difference map 409 is analyzed to determine whether any such incompatibility is sufficient as a possible indicator of the target 105, and to determine whether these standards are in line with the camera's field of view (205 or 206). The positions match. The position of the target 105 can be further fine-tuned in block 411 (as needed), which generates the camera-related manifestation and position related to the target 105-16-This paper standard applies Chinese National Standard (CNS) A4 Specifications (210 X 297 mm) 543323 A7 B7) 5. Description of the invention (14 sets output 310 or 311 (as described in Figure 3). In Figure 41, box 4 02, 'Image processor 1 06 extraction and activities The image data corresponding to the image area 208 (Fig. 2B). The image data can be retrieved by cutting, cropping, rotating or transforming the captured image data. The removal only captures one of all the images in the active image area 208 Part. Define the boundary, and copy all the pixels inside the boundary to the new buffer, while ignoring the pixels outside the boundary. The moving image area 208 can be any shape. Crop and rotate to reorder the data In order to facilitate further processing of the sequence, such as rectangles, it can be addressed in the ranks of pixels. Rotation processing makes the displayed image content as if the image has been rotated. Rotation processing makes the pixel position according to the following equation ( x, y) are then sorted into (x ,, y,):-/ cos0-sin0 〇 'X / y sin0 cos0 〇y _1_ 0 〇1_ 丄 where θ is the image rotation angle. For example, cameras 101 and 102 are opposite It is fixed in the target area 103 correctly, and the angle to be rotated is generally not large. If the angle to be rotated is not large, the front crop provides a simpler calculation to drive closer. The crop will distort the shape of the image, causing Transformation (The shape looks like the columns and rows have been sliding up and down each other. Reorder the pixel positions according to the following equation: aji 543323

AT ______B7 V. Description of the Invention (15) where shx is the horizontal trimming amount in the image, and shy is the vertical trimming amount in the image. In the context of applying the multiple camera control system 100 to the target 105, the whole or part is not rounded, similar to illumination with a higher or lower control background 104. For example, the situation can be created by lighting the background 104. The filter box 40 passes the lighting information related to the image data. The single background mode 406 is the background 104 lighting as expected. In fact, the lighting of the controlled background ιo4 can be changed in the moving image area 208, so the background mode 406 can be stored in the moving image area 208 for the expected lighting value for all pixels. The comparison standard generates and processes 4 〇1 No. 1 noise (the above can be considered in the background mode), and by modifying each lighting value from the background mode 406, the lighting of the controlled background ι04 is changed slightly, thus generating a small amount Illumination value, which can be classified as compatible with the background mode. For example, if the lighting of the controlled background 104 is higher than the lighting of the target 105, the processing block 407 will reduce the lighting value of each pixel by a value greater than the desired signal and the lighting changeability. In some implementations of the system 100, the target area 103 is narrow enough to allow it to be modeled as a planar area. The turning of this plane is parallel to the front and back of the foot cube of the target area 103 shown in Figure 1. In the optional scale module 视 4, the moving image area 208 can be reduced to a single row of pixels. If the following two conditions are met: 1) When the target 105 is detected, some rows in the moving image area 208 will bear the background 1G4 in all columns, and 2) a single set of values in the background mode 4Q6 is sufficient Describe the characteristics of all the rows of pixels in the moving image area 208, then in the scale module 404 as needed, the moving image area can be reduced to the pixels in the early column. If the moving image area 208 is thinner than the target 105, the first item 543323 A7 B7 5. The description of the invention (16) is often consistent. The second condition is met by the execution of blocks 403, 405, 406, and 407. The application of the scale module 404 can reduce the processing complexity required when performing subsequent processing, and can reduce the storage requirements of the background mode 406. The specific implementation of the scale module 404 is related to the characteristics of the processing blocks 403, 405, 406, and 407. If the illumination of the controlled background 104 is expected to be higher than that of the target 105, as described above, the implementation of the scale module 404 represents each row with the maximum illumination value in the row. This process has the added benefit that the highly illuminated portion of the controlled background 104 need not fill the entire controlled background 104. Another condition applied in the context, where the controlled background 104 is stationary, i.e. does not include motion, but is not limited in lighting. Take the sample source image contained in Figure 5A as an example. In this case, the target that is the sensing side of the camera may include or be closed to the size of the lighting value, which can also be seen in the controlled background 104. In fact, the magnitude of the change in the illumination of the controlled background 104 (for example, caused by the user moving in front of the device, thereby framing some ambient light) is significantly related to the difference between the controlled background 104 and the target 105. Therefore, a special type of filter can be applied to the screening process 403 to produce a result that is irrelevant to the overall illumination or has a low degree of variability, while highlighting a portion of the target 105. A 3x3 Prewitt filter is generally used in the filtering process 403. Figure 5B shows the results of the screening process 403 on the image of Figure 5A. In this implementation, the two background modes 406 can be maintained, each of which represents the high and low values, and collectively represents the range of values expected for each of the selected pixels. The comparison standard generation process 407 then decreases the low value and raises the high value by an amount larger than the expected signal noise and lighting changeability. The result is a set of standards. An example of the low value is shown in Figure 5C. For the high value, -19- This paper size applies the Chinese National Standard (CNS) A4 specification (210 X 297 mm) 543323 A7 _-_________ B7 V. Description of the invention (17) IIij is shown in FIG. 5D. These modified images are processed by comparison * 〇 $, if the value is lower than the low value standard (Figure 5C) or higher than the high value standard (Figure 5D), it classifies the pixels as different from the controlled background 104. The result is a binary difference graph 409, an example of which corresponds to FIG. 5β, and is shown in the graph. Previous implementations allowed many existing surfaces such as surfaces, walls, or window frames to be used as the controlled background 104, where these surfaces It can have any illumination, organization, edge 'or even the light band fixed to the controlled background 104. The above implementation also enables the use of a controlled background 104 including a predetermined pattern, or tissue, and light band, among which the above processing detection In the controlled background 104 is a pattern that is short of the target 105 blocking area. The difference map 409 stores all pixel positions found in the above method that are incompatible with the controlled background 104. In this implementation, The difference map 409 can be any image table, in which each pixel can be in one of two states. The pixels that are not compatible with the controlled background 104 are set in the corresponding column or row of pixels in the difference map. Wait for one of the states to recognize or deteriorate. The implementation of the measurement module 410, wherein the detection of the target 105 in the difference map 409 is shown in Fig. 6. Another size module at block 603 provides an opportunity to reduce the data to one-dimensional array data, as needed and The turn that can be applied to the target 105 has no significant effect on the overall boundary of the target 105 in the difference chart 409. In fact, this is applied to many columns with a number lower than or similar to that of the target 105 In the context of a typical number of rows. At the time of application, the size module of block 603 reduces the difference map 409 to a single-line diagram, that is, a one-dimensional array value. In this implementation, the size module 603 can calculate the difference The number of pixels marked in the row of Figure 409. For example, the difference chart 409 of Figure 7A can be -20- This paper size applies the Chinese National Standard (CNS) A4 specification (210 X 297 mm) 543323 A7 _ __B7 V. Invention Explanation (18) The method reduces and draws the image 709 as shown in Figure 7B. This additional processing step is applied to reduce processing requirements and simplify some subsequent calculations. Continue the implementation of this detection module 410, and observe the difference shown in Figure 409 (E.g. FIG. 7A) related to the target 105 Pixels will generally form cluster 701, but cluster swapping needs to be linked. Cluster identification processing 604 classifies pixels as whether they are a member of cluster 701 (or if the size module 603 has been applied, the rows are classified ). There are multiple sampling and clustering methods that can be applied, and the subsequent method has been selected to simplify the process. Note that when the target 105 appears, the number of correctly labeled pixels will exceed false-positive (false- p〇sitives) number. Therefore, the middle position can be expected to fall somewhere within the range of the target 105. Part of the cluster identification processing 604 is performed when applied to a single-line diagram (for example: the size module in box 603 or 404 has Where it is applied), it calculates the middle row 702 and the marked behavior part cluster 701 (FIG. 7B) (if it is within a predetermined distance 703 corresponding to the highest number of rows occupied). Implementation of the partial cluster identification processing 604 'When applied to a multi-column map, if it conforms to the proximity-distance criterion, the marked pixels are added to the cluster 703. In this implementation, a set of criteria is received by the cluster classification process 605, and then the cluster 701 is used to confirm that the cluster is sufficiently consistent with the desired target ι05. Therefore, the process 605 determines whether the cluster 701 will be classified as belonging to the target 105. The implementation of partial cluster classification processing 605 is to calculate the number of labeled pixels in the cluster 701, and calculate the total number of labeled pixels. Compared with the threshold in the cluster 701, the number of pixels marked in the cluster is eliminated, which is regarded as the non-conforming part of the target 105. The number of pixels in the cluster 701 is compared with the total ratio and the threshold value to further reduce the inconsistencies. -21-This paper size applies to Chinese National Standard (CNS) A4 (210 X 297 mm) A7 B7 543323

19) If cluster 701 passes these standards, the description of the cluster is fine-tuned in block 606, which is calculated from the center of gravity associated with cluster 701 in processing 607. Although the intermediate position found by the size module 603 may be within the boundary of the target 105, it does not need to be located at the center of the target. A weighted average of 7 1 0 or weight / ν > is used for better measurement of the cluster position ' and may be calculated in the processing Mg as the secondary processing 607 if necessary. The weighted average of 7 1 0 is calculated by the following equation:

Hold

ICM jc =: 0 Where: ί is the average value C is the number of lines C [x] is the number of marked pixels in the X line 玎

The cluster boundary 704 can also be calculated in the process 606 as needed, as shown in the process 608. The cluster 703 may contain some false-positive separators. As part of this, the boundary may be set around a certain percentage of the labeled pixels, or in the situation where relatively few labeled pixels are expected. Surround the marked pixels (or lines, such as when the size module 603 is applied) to form a close sub-cluster, that is, the marked pixels (or lines) that are also marked near Zheng. Except for the central and boundary coordinates, the direction of the target object can be estimated from the cluster momentum calculation as required. This aspect is based on the cluster aspect in the second place within the processing of 606: 609 to calculate the processing table. In some applications of the system 100, the target 105 is used as an indicator. Here m ′, if the target area 1 () 3 contains a sufficient number of columns and the number M is not reduced, then it can be expected and the pointing end of target ㈣5 can be within -22 of ㈣6.

543323 A7 B7 V. Description of the invention (20) The calculation of the secondary end is determined by the pointing end. An example continues as shown in Figure 7C. Generally speaking, the target 105 will enter from the boundary of the area or enter the moving image area 208 due to restrictions. The pointing end 705 of the target 105 (for example, the finger end of the user) may be the part of the cluster 70 1 farthest from the entering area 706 entering the moving image area 208. Cluster 701 may contain some false-positive isolates. In this way, the pointing end 705 may be defined by the area 707 within the cluster 701, which surrounds the multiple labeled pixels near the farthest boundary side of the cluster 701, or, in the situation where relatively few pixels are marked, The farthest marking pixels that form a close sub-cluster; that is, the marking pixels that are also marked in these neighborhoods. This cluster system is identified by the secondary cluster pointing end processing 6 10 and the location of the secondary cluster known in processing 611. Continuing with this process, a smoothing module 612 may perform a process as needed, and apply it to any or all locations known in process 606. Smoothing is a process that combines the results of previous solutions to move it from frame to frame in a stable manner. The center of gravity determines the weighted average coordinate 7 1 0 obtained from processing 6 07, which is related to many samplings and is therefore naturally stable. The cluster boundary size determines the boundary 704 obtained by processing 608, and the pointing end 705 obtained by 611 is related to relatively few cluster groups, and a single pixel state can have a significant impact. Since the size of the area occupied by the target 105 is expected to remain relatively stable, smoothing can be applied to the distances between the boundaries 704 measured relative to the cluster weighted average coordinates 71. Since the desired target 105 changes relatively slowly to the overall position of the target 105, the smoothing can be applied to the pointed end 705 as measured relative to the cluster weighted average coordinate 7 10. distance. The processing adopted at the center of gravity processing 607 is the following equation i: -23-

543323 A7 ___ B7 V. Description of the invention (21) s (t) = (axr (t)) + ((ia) xs (tl)) In Equation 1, the smooth value at time t (s (t)) Equal to one minus the scalar value (a) times the smooth value at time minus one (M). This quantity is added to the column value at time t (r (t)) times the scalar value (a) between zero and one. Referring to Fig. 8, the implementation of the system 100 utilizes one or more of the background modes 406 (Fig. 4). The execution or generation of a background pattern processing component 405 of the background pattern data 406 is shown in FIG. 8. The execution of this background mode processing component 405 automatically generates and dynamically updates the background mode, so that the system's unaccompanied operation can be performed. The input data 802 is provided by the execution of the background module processing component 405 by the scale module 404. Enter each time frame available and sample in the sampling process 803. Sampling may include a target 105 that blocks part of the controlled background 104. For each voxel, a range value can present a better background 104 than a single value. By incorporating this range effect into the background model, the expansion in processing 407 can be made even closer. Contribute multiple time frame data to the sampling so that this range is visible. 'However, at the time of sampling, if the target object 105 is moving, the portion of the background 104 which is supported by the target object 105 will also increase. The optimal value of the time frame used depends on the expected movement of the target 105 in the special application of the system. In fact, for a system that searches for one hand, 10 o'clock, and presents for about 0.33 seconds, it is sufficient to observe most of the range without having to block the target excessively against the background part. If the special background mode is compared with the comparison process 408 and the upper boundary of the value is regarded as compatible with the background 104, then the maximum value of each pixel observed in the multiple time frame can be recorded as the sample value. If the comparison of the special background mode 4106 is processed 408, and the lower boundary of the value is regarded as compatible with the background 104, then -24- this paper size applies the Chinese National Standard (CNS) A4 specification (210 X 297 mm) 543323

AT B7 V. Description of the invention (22 The minimum value of each pixel observed in the multiple time frame can be recorded as the sampling value. In the execution of the background mode processing component 405, the sampling of the self-sampling processing 803 is added to the storage location. The buffer 804 俾 stores n samples, in which the last sample is replaced in this history. Therefore, this history contains n samples for each pixel. The time interval d in the buffer indicates the rate required for the new samples. Set 'and add it to the history r via the following Equation 2: In this implementation, the intermediate processing block 805 selects for each pixel its decision value at the position represented by the prime and the controlled background 104. One kind of processing block The method used in 805 to select the representative value of the controlled background 104 is to select the intermediate value of the n samples of each pixel. For any pixel, multiple n samples in the buffer 804 can represent the target object. 105. The time interval d is selected so that if it does not add up to d / 2 in any time interval d or longer, the target 105 will block any of the controlled backgrounds 104. Pixels, so for any pixel, most of the The sample value will be shown in the background 104, and the median sample value will be the value shown in the background 104. The processing component 405 of the scene mode can be adjusted, as long as the viewing time is d / 2, Any changes to the background 104 will be reflected in the output of the intermediate processing block 805. At the beginning, this system does not require the entire controlled background i04 to be visible. At the beginning, the target 105 may exist, but before the output is provided, Sampling needs to be observed at time d. If necessary, the target object 105 can be restricted from existing at the start of the system. In this case, the first observation sampling value can be copied to all n samples in the buffer 804, so that This system generates output quickly. Any pixel period of the controlled background 104 will be blocked by the target object ι05, and because of -25- this paper size applies the Chinese National Standard (CNS) A4 specification (210X 297 mm) 543323 A7 _____ B7 V. Description of the invention (23) At this time (1 is an unsystematic drop, it makes -rr Ding Jizhumu application called 疋. The number of samples ri can be adjusted to the memory buffer, and can be processed in series. In previous discussions, the presentations were made by cameras 101 and 102 The image is related to or obtained from the position of the target object within its range of 105. If the target object 105 is successfully detected, and its coordinates can be detected by the detection modules 3008 and 3009 in Fig. 3 in the field of view of the two cameras If found in 205 and 206, these coordinate groups are sufficient to restore the target 105 position within the range of the target area 103. In the implementation shown in FIG. 3, the target 105 position is calculated in the combination module 312. Turning to FIGS. 9A and 9B, the illustrated system is implemented by one of the combination modules 3 and 12. For each of the cameras 101 and 102, the position P 902 of the target 105 on the camera image surface 904 is converted to an angle 905, In this description, the cold table is used, which is the measured value on the reference plane, and the normal is determined by the rotation axis of the camera 01, 102. (Actually, the axes are not exactly parallel but on a single plane, but the processing described here is within error tolerance). By approaching the cameras 101, 102 as the ideal pinhole pattern of the camera, the relative angle from the camera's amplitude to the predetermined vector 906 can be cooled. As shown in Equation 3 in Equation 3, the approach calculation is as follows: f β = tan'1 — [pj is the temperature approaching this angle, and the inverse tangent is applied to the focal length (f) divided by the projection on the image plane to the reference The amount of position P on the X boundary between the plane and the image plane. For maximum accuracy, the essential camera parameters (principal point position and image size) and the radial distortion due to the lens should be modified. This is achieved by converting the distortion position (from the relative position information 310 and 311) to the ideal position as Of it. You-26- This paper size applies the Chinese National Standard (CNS) A4 specification (210 X 297 mm) 543323 A7 _____B7 Description of the invention (24) ^ ^ — such as camera 10 1 and 10 2 have ideal pinhole camera properties , The ideal position is the position where the object 105 is projected on the image plane 904, and an appropriate angle can be generated by Equation 3. Z. Zhang in "A nexible New

A set of correction procedures proposed in Technique f0r Camera Calibration, Microsft Research, httP://reSearch.miCr0Sft.com/~zhang ,, and is hereby incorporated by reference. For many applications of this system, it is known that this approach requires the above corrections to have sufficient accuracy. Continuing the description of the combination module 312, as shown in FIG. 9B, a reference vector 907 is defined to pass through two cameras and positions on the reference plane, where the definition of the reference plane makes the camera's corner determine the normal of the reference plane. The camera rotation angle 908 is measured with respect to the reference vector 907. A formula for measuring angles is shown in Equation 4. a = β 〇 β The measured value of the angle α is equal to the angle yj. And the horn cold sum. Equation 4 is applied to measure the angle 909 between the target 105 and the reference vector 907. The angle is expressed in α here. The angle α 909 and the length of the reference vector 907 for the cameras 101 and 102 are sufficient to find the position of the target object 105 on the reference plane from equations 5 and 6.

Equation 5 can calculate the offset of the target —wtanaAtanaB

tanaA + tanaB partiality (y) is equal to the angle (aA) of the camera A ιοί and the reciprocal of the sum of the tangent of the camera 102 102 < angle (aB) times the vector length 907 (w), to camera A 1 The angle (aA) tangent of 1 and the angle (aB) tangent to the camera B 102. -27- The paper or the scale is in accordance with the Chinese National Standard (CNS) A4 specification (2ι〇χ mm) 543323 A7 __ B7 _ 5. Description of the invention (25) Equation 6 can calculate the target offset (χ A) as follows: y xA =-

tanaA In Equation 6, the offset (Xa) is calculated by dividing the offset (y) of Equation 5 by the angle (aA) tangent of the camera A 101. The position of the target 105 on the axis perpendicular to the reference plane can be obtained from Equation 7 ', which uses the distance between the target 105 and the camera to apply the position in each image. z = l — f's position (z) in Equation 7 is calculated as the position on the image plane &) Projected onto the image plane vector perpendicular to the plane used by Equation 3, divided by the focal length (f ) Multiply the distance between the target 105 and the camera (1). These relationships provide the coordinates of the target 105 relative to the camera a: ^ 丨. Knowing the position and size of the target area 103 with respect to the camera A 101, the coordinates can be changed so that it is opposite to the target area 103, as shown in Fig. 3-12. If necessary, smoothness can be applied to these coordinates in the execution trim module 313 of the system shown in FIG. Smoothing is a process that combines previous results to make the frame move steadily from time frame to time frame. A method for smoothing these specific coordinate values (χ a, y, z obtained from the combination module 3, 2) is described here. The components of each coordinate value group are related to the target 105, that is, X, y, and z are independently and dynamically smoothed. The degree of suppression S is calculated by Equation 8, where S is dynamically and independently adjusted in response to a change in position. The calculation is as follows: -28- This paper size applies to China National Standard (CNS) A4 (210 X 297 mm) 543323 A7 B7 V. Description of the invention (26 S = ^ (l-cc) SA S8 where a =

Ds-Da if {D < DA) M〇a &D; D < Db) if {D > D3) βΗκ,)-rush-1) 丨 In Equation 8, s (t) is the smooth value at t, ^ It is based on the "natural value of the temple, DA and DB are critical values, and, and, and define the degree of inhibition. The two distance critical values DA and DB, as shown in Figure 丨, define three ranges of movement. Position change is less than Da , Then the motion system is severely suppressed by%, which reduces the tendency of a value to switch back and forth between two close values (the second effect of image separation sampling). If the position change is greater than Db, the motion system is slightly reduced by% Inhibit 1002, or not be suppressed. This can reduce or eliminate the delay and blur state introduced in some other smoothing programs. The degree of inhibition of motion varies between Da and db, and its area is marked as 1003, making The transition between mild and severe suppression is less pronounced. The quantity a applied to equation 丨 can be seen in equation 9 as follows:

The amount (a) of S in the formula 9 is limited such that it is equal to or greater than zero and less than or equal to one. The suppression value S is found in equation 8, and e is the elapsed time from the previous time frame. If these targets 105 coordinates 314 can be found, they are typically transmitted to another process for use, such as the user application 3 16. It can be transferred to another process' which performs the same calculations as described above performed by the image processor 106 or to another computing device. Send the data to the application program here. 29- 543323 V. Description of the invention (27.16 million methods, which can include the traditional user input and keyboard) make the system and use the private type in the deer " '^, ... Control of existing control functions in the range of 316. It can be used for all the coordinates 105 of the video image objects captured by the camera with money, including 314, of which the video frame is typically captured 30 times or 4 times per second. Causes a slight latency between user action and application response. In a typical application of the system, the 'application 316 has user feedback, and the visual representation of the-indicator is displayed on the photographic display screen. Make the indicator move 'so that its position and movement are similar to the movement of the target 105 (the point is the user's hand). In the variation of the user interface type, this indicator, such as a mouse cursor, is shown in front of other images' and its movement is reflected in the two-dimensional space defined by the surface of the curtain. This type of control is similar to computer mouse functions, such as those used in Microsoft Windows operating systems. _Application feedback image example uses control pattern 1102 as shown in Figure 11A. Referring to FIG. 11A (as shown in FIG. 3 for simplicity), the image processor i 〇6 also includes a coordinate re-processing 317 (FIG. 3) as needed. This coordinate then responds to the processing of the 317 series operation and re-maps the overall presentation and position coordinates 314 (related to the target 105) to the position where the index 1101 (such as a cursor or mouse pointer) overlaps on the image 1102, using Equation 10 For the X coordinate and the equivalent equation of this equation ^ The second to the y coordinate is as follows ... as xh < bt one · as bt-xh 幺 K br -〇 1 as xh > br -30- 543323

AT __ B7 V. Description of the invention (28)-In Equation 10, 'X h is the coordinate position related to the target 105; > 1 4' X e is the position of the screen upstream index, which is mapped to 0.1 B ^ and br are the left and right borders of the sub-region within the range of the target region 103. As shown in Figure 丨 丨 B, the entire display area 1102 is represented by the secondary area 1103 which is completely contained in the target area 103. The position within the sub-region 1103 (for example, position A 1105) is linearly mapped to the position within the display 1102 (for example, 1106). A position (for example, position B 1107) outside the sub-region no3 but still within the target area 103 is mapped to the nearest position (for example, 1108) on the boundary of the display area 1102. This can reduce the possibility that the user may unexpectedly move the target 105 (usually the user's hand or the index finger) from the sub-region, while trying to move the index 1101 to a position close to the display boundary. In the situation where the target area 103 is before the photographic display 107, the sub-area 1103 may be aligned with the photographic display 107, so that the index 1101 will appear to be aligned with the target 105. If the target area 103 is relatively thin, for example, less than 5cm, and the subarea 11 03 is defined in this way, the system approaches the screen with user interaction, not limited to the photographic display 1007, and There is no need for direct contact between the surface of the photographic display 107 and the user (for example: the photographic display and the user can be on the opposite side of the window). It will be understood that the system 100 can be used with various sizes of display screens, and can include not only computer monitors (whether CRT or LCD type monitors), but also rear-projection TV monitors' large flat-screen LCD monitors. And a front projection display system. In the situation where the target area 10 3 is not immediately before the photographic display 1 07, and the moving image area 2 08 of the target object width direction found in the width direction calculation processing 609 is deep enough, a vector can be drawn from the target object It extends to the photographic display, which uses the vertical angle to detect the user's pointing position on the photographic display. -31-This paper size applies to China National Standard (CNS) A4 (210X297 mm) '---------- 543323

AT ___ B7 V. Description of the invention (29) But the most common one is that the depth of the moving image area 208 is not enough to accurately calculate the amplitude direction in the processing block 609. In these situations where the target area ι03 is not immediately before the photographic display 107, and the width direction cannot be calculated, the equation 1 can be applied in the sub-region 1103 which is smaller than the photographic display. Then the processor maps the absolute position of the target object 105 to the position indicator, so that the movement of the target object ι05 is adjusted to a large position on the photographic display, and the entire photographic display area is easy for the user (For example, the width of the sub-region 1103 can be set to 750 mm at most and proportional to the height, and its size is accessible to most users). When set in this way, this system still gives the user the feeling of pointing at the screen. In another variation of this type of user interface, the user moves the indicator in a three-dimensional virtual reality (examples are shown in Figures 2A and 12B). This virtual reality can be rendered using a projection transformation, so that the depth of the virtual reality includes the image presented on the photographic display 107. Techniques for segmenting such virtual reality include OpenGL. Equation 10 is used to remap the X, y, and z coordinates (for example, the subregion 1103 becomes a cube). The application is controlled by a movable on-screen indicator (for example: Figures 1A, 12A, and 12B). The control in it has been discussed before. Typical presentation of data image representation or interactive components (for example: button 1109 or object representation 12). 2). The user expects to place the index 1101 on one of these objects, or to contact or interact with the object if a three-dimensional virtual reality is presented. For a two-dimensional interface, this condition can be detected by comparing and re-mapping the position of the indicator, which is 1060, and the object image representation boundary (for example, 1110), and if the indicator position is within the boundary of the object, then this condition Is true. For 3D interfaces, this situation can be compared by -32-

543323 A7 _ B7 V. Description of the invention (30) The overall index 1101 or the finer control is the boundary 1204 of the partial index boundary 1203 and the object 1202. The user may receive feedback indicating that the cursor is positioned on the object as needed. The feedback can be of various types, including audio prompts and / or changes in one or both of the cursor and the object's image representation. The user can then start, copy or move the object under the cursor. The user can make a gesture to start, copy or move the object. The movement of the target 105 can be translated and classified as gesture detection processing 3 15 as needed, as shown in Figure 3 above. Gesture detection processing 3 1 5 can use data generated from any part of the system. The combination of the final coordinate 314, the image coordinate 310 and 3 11 or 3 10, 3 11 and 3 14 can be sampled over time and used as input to the gesture detection process 3! 5. Various gestures (for example: Coverage and Good can successfully use this data to detect the input to gesture detection processing 3, 5). In a situation where the application status is known and transmitted to the gesture detection module 3 丨 5 (that is, whether the indicator 1101 is overlaid on the button 1109). A gesture used by a user to indicate that an object (eg, screen objects 1109, 1202) is not to be activated under the cursor 1101, causing the cursor to hover over the object (eg, 109, 1202) for a longer time than a predetermined duration . The detection of the gesture performed by the user is to detect the application state and trigger the gesture when the application state remains unchanged for a predetermined period of time. It is not necessary to generate this application specifically for the multi-camera control system 100. Existing technology can be used to monitor the application status at will (using the Windows SDK function "etWindowsHookEx" in the Windows operating system to set an ook), as well as flicking with the mouse (in In the Windows operating system, the use of the Windows SDK function "endlnput ,,") ° In some scenarios, 'application status may not be obtained and may not be monitored -33-This paper size applies the Chinese National Standard (CNS) A4 specification (210X 297 mm) 543323

AT ____B7 V. Description of the Invention (31 ~) 'Control. In this case, some exemplary gestures (eg, screen objects 1109, 1202) to indicate the object to be activated under the cursor 1101, to maintain a stable hand coverage or quickly fiddle with the hand. A method of detecting coverage is to save the position history of the target object 105, where the history includes all position and status records within a predetermined period of time, and ends at the most recent sampling. This period is the minimum period during which the watch user must keep his hands still. The minimum and maximum positions in three dimensions (X, y, z) can be seen in history. For example, in all samplings in history, the target object 105 appears in the target region 103, and the distance between the minimum and maximum values is within a three-dimensional predetermined threshold value ', then the coverage gesture is reported. These distance critical values are the maximum and maximum changes that the target 105 can allow, or it can be introduced into the hand position by various components of the system. This gesture is reported in a typical method, where the system mimics the same movement as described above, and mimics the same gesture as the tap. Gestures represent additional operations of gas-pressing, double-clicking and pressing, and they are similarly detected after being detected. In addition, if necessary, gestures that are not related to the position of the indicator relative to the object can be detected and given meaning with applications that may or may not be related to the application status. Applications that take advantage of this type of interaction typically do not explicitly use or display the target location 3J7 or other locations. These applications as a whole or mainly are controlled only by the position interpreted by the system. These applications also do not need to be created specifically for this system, so the system can be used to simulate actions performed on traditional user input devices, such as a keyboard or joystick. The useful solution of Dudu depends directly on the absolute position of the target object 105 in the target area i 03 (or the index position in the sub-region 1103 can be used in an equivalent manner). One way to make this interpretation is to define a box, a flat surface, or -34-This paper size applies the Chinese National Standard (CNS) A4 specification (210X297 public love) 543323 A7 _____B7 V. Description of the invention (32 'its i shape. A state To trigger, for example, find the target 105 position in the first box (or outside the boundary defined by the first plane) (for example, the position defined by block 314, or co-ordinates since re-processing 317), and its It does not exist in the observation just now (it is not in the area outside the target area 103 or is not detected). This state is maintained until the position where the hand is not found in the second box (or is defined by the second plane Outside the boundary), the state will be closed in due time. The second box must contain the entire first box, which is typically larger. Use of a larger box can reduce the undesired state when the target 105 is detected near the box boundary. Trigger or close, where a small movement or a small amount of image signal noise will cause position 3 17 to drift inside and outside the box. Typically, one of the two methods is used to resolve the sadness of the gestures you want to use. In one method Gestures directly reflect the triggering or closing state. When the button is fired like the same keyboard key or joystick, press in the triggering state and relax in the off state. In another method, the gesture only transitions the complaint from off to on. When Simulate the same keyboard key or joystick button, press the key. Although the period is not reported and the closed state is applied to the application, the gesture will not be repeated until the state is closed, so that each situation of the gesture must be clearly defined by the user. The third method is to trigger a gesture when the state transitions from off to on, and then trigger the Ding potential periodically at a predetermined period to keep the state on. This is similar to keeping a key down on the keyboard so that the characters are in the It is repeated in some applications. In one of the above-mentioned technical methods, the box or surface can be defined within the target area 103. It is described as follows. By defining the first surface (ι501) in FIG. 13A and the first surface 1502 'Segment the target area into the firing area 1503 and the central area 1504 (when the target 105 is in the area 1505 between the planes, report the hand based on the previous position of the object -35-

543323 A7 _____B7 V. Description of the invention (33-) '' " ~ The above-mentioned technology can detect the target 105 pushing forward (typically a hand), and the child gesture is used to imitate the same as on the joystick. Fire a button, or cause the application to respond in a manner that is related to pressing the joystick button (such as a weapon fire in a virtual game). In another above-mentioned technical manner, the box or surface may be defined in the target area 103, which is described as follows. The first type planes 1506, 1507, 1508, and 1509 are determined to divide the target area 103 into left, right, upper, and lower portions, and overlap the active area, as shown in FIG. 13B. The second line of the plane is labeled 1510, 1511, 1512, 1513. Each of the first and second planes is treated independently. This plane group is similar to the four direction cursor keys, in which the two keys are triggered by the hand in the corner, which is generally interpreted as four auxiliary 45-degree (diagonal) directions for the final multi-application. In this method, the pseudo-keyboard cursor makes various existing applications controlled by the system 100, including, for example, Microsoft Power Point, which responds to the movement of the pseudo-cursor keys (such as the up and down arrow keys) in the presentation sequence. Next or previous slide. Another common analog control method applicable to the application is to accurately display the four 45-degree directions. Boxes 1 5 14, 1, 5 1 5, 15, 16, 1 5 17 are based on the four main directions (horizontal and vertical), while the additional boxes 1518, 1519, 1520, and 1521 are 45 degrees to the four auxiliary ( Diagonal) direction, as shown in Figure 13C. For brevity, only the first type of box will be described. Place a gap between these boxes. FIG. 13D illustrates how to locate adjacent boxes. The gap between the first type boxes 1522 and 1 523 can ensure that the user intends to let the target 105 enter the box, and at the same time, the gap 1524 is filled by overlapping the second type boxes 1 525 and 1526, so that the system reports the previous gesture until use The user clearly wants to move the target 105 into an adjacent box or a central neutral zone. This button group can be used to imitate the same eight directions. -36- This paper size is applicable to the Chinese National Standard (CNS) A4 specification (210 X 297 mm) 543323 A7 B7 5. The joystick key of the invention description (34). The broader class of gestures depends on motion rather than or in addition to position. An example is 爿 亭 + the gesture of swinging to the left. This is a gesture that is communicated to the app that it wants to return to the previous stomach or state. This gesture can be used to control information presentation software (especially Microsoft PowerPoint) to the previous slide in the presentation sequence by using the same keys and mouse. This gesture allows the web page to perform the actions associated with its reply key, using the same keys and mouse. Similarly, swinging the hand to the right # is a gesture used to communicate to the app that it wants to go to the next page or state. For example, this gesture causes the rendering software to go to the next slide in the presentation sequence and causes #browser software to switch to the next page. A method for detecting hand swing to the left is described as follows. A thin strip along the left-most part of the heading area 103 is designated as the left margin area. The position of the target 105 (for example, the position set by the square 3 14 or the re-coordinates determined by the self-reaction processing 3 17) is shown in the following three states: 1 The target appears but is not in the left margin area. 2. The target appears and is in the left margin area. 3. The target does not appear in the hand detection area. The above transition from state 1 to state 2 causes the gesture detection module 3 15 to enter-71 due to complaints', thereby starting a timer and waiting for the next transition. Such as the transition to state 3 is observed within the predetermined period, and the hand gesture of reporting to the left is reported. This technique can generally be repeated on the right, upper, and lower edges, and because the position of the hand can be seen in three-dimensional space, it is also copied to detect pull back. Various gesture detection techniques have been discussed. In the research literature, there are other gesture detection technologies (for example: Hidden Markov Layers), which are applicable to this -37- This paper size applies to China National Standard (CNS) A4 (210 X 297 mm) 543323

AT _____ B7 V. Invention Description (35) Various implementations of the system 100 are described. Referring back to FIGS. 1 and 3, the implementation of another multiple camera control system 100 is described in more detail. Even with the dual camera system shown in Figure 1, it should be understood that the image processor 106 may be configured to receive input from more than two cameras' in special applications and may include four or more camera cameras. In the implementation of the four cameras, the parts 30-311 of Fig. 3 are reproduced to support the two additional cameras. In addition, the combination module 3 12 is configured to receive and search four sets of camera-related targets 1 05 related presentation and location data (similar to data 3 10 and 311). The aforementioned techniques and equations (especially equations 5 and 6) are applicable to additional camera pairs, where the output of the combination module 3 12 is the average value from all positions of each camera pair. The configuration of the gesture detection module 3 1 5 is similar. From the two major detection modules similar to the detection modules 3 10 and 3 11 (similar to 3 0 8 and 3 0 9), it receives four sets of cameras and cameras. Relevant presentation and location information 310, 311. The output from the image processor 106 currently contains processed object position coordinates and gesture information related to the four cameras, which can be used by another process or user application software 316. The formula and geometry (above) used to calculate the coordinate information related to the target 105 from the two additional cameras are available. In an implementation using four cameras, two additional cameras are located on the two corners below the controlled background 104, and are rotated so that the target area 103 is within the field of view 205 of each camera. The advantage of the four-camera system is that the position search accuracy of the target 105 is high. Therefore, the application can include more high-density screen objects on the photographic display 107. This is because the search accuracy is increased, the objects can be closer, and the correct selection of Mumu ^ -38- This paper size is applicable China National Standard (CNS) A4 specification (210 X 297 mm) 543323 A7 B7 5. Description of the invention (36) Some slight movement of the object 105. In addition, when a part of the object 105 blocks the field of view 205 associated with one or more other cameras, two additional cameras can reduce the error in searching for the object 105. Although various implementation methods have been described, it should be understood that various modifications can be made. According to the above, other implementation methods fall within the scope of the following patent applications. -39- This paper size applies to China National Standard (CNS) A4 (210 X 297 mm)

Claims (1)

543323 Copper see No. 090124363 Patent Application Chinese Patent Application Replacement (April 1992), Patent Application Scope 1. A method for tracking a target, the method includes: obtaining a first image and a second image representing different views of the target · Process the first image into the first image data set, and process the 2nd 7'th image into the second image data set; process the first image data set and the second image data set to generate background-related background data 〃 Generate a first difference map by determining the difference between the first image data set and the background data set, and determine a second difference map by determining the difference between the second image data set and the background data set; Measure the first relative position of the target in the first difference map and the second relative position of the target in the second difference map; and the absolute positions of the first and second relative positions from the target that are generated from the target. 2. The method according to item 1 of the scope of patent application, wherein the steps of processing the first image into a first image data set and processing the second image into a second image data set include determining a moving image area of the first and second images , And the first and second images in the moving image area to capture the moving image data set. 3. The method according to item 2 of the patent application range, wherein capturing the moving image data set includes cropping the first and second images. 4. The method according to item 2 of the patent application range, wherein capturing the moving image data set includes rotating the first and second images. 5 · The method according to item 2 of the scope of patent application, wherein capturing the moving image data set includes trimming the first and second images. This paper size applies to China National Standard (CNS) A4 (210 X 297 mm) 543323 Patent application scope AB c D 92, year 01 * 6 · If the method of the second patent application method is applied, which captures moving image data The set includes a moving image data set configured as an image pixel array with columns and rows. 7. The method according to item 6 of the scope of patent application, wherein capturing the moving image data set further includes identifying the highest pixel value in the range of each image pixel array line; and generating a data set with a row so that the highest value identified by each line The pixel values refer to this row. 8. The method of claim 1 in claim 1, wherein processing the first image into a first image data set and processing the second image into a second image data set includes filtering the first and second images. 9. The method according to item 8 of the scope of patent application, wherein the screening includes expanding at the edges of the first and second images. 10. The method according to item 8 of the scope of patent application, wherein the screening further includes processing of the first image data set and the second image data set, emphasizing the difference between the first image data set and the background data set, and emphasizing the second Differences between image data sets and background data sets. 11. The method of claim 1 in the scope of patent application, wherein the processing of the first image data set and the second image data set generates a background data set, including generating a first set of one or more sets of backgrounds related to the first image data set A data set, and generating a second set of one or more sets of background data sets related to the second image data set. $ 12. The method according to item 11 of the scope of patent application, in which the background data set of group _ or group xi is generated, including the first image data set representing the background ^ -2-, the scope of patent application ^ produces the largest representative data The first background set of values and a second ::: or multiple sets of background data set are included in the range of the second image material set representing the background to generate a second background set representing the largest data value. 3. The method according to item 12 of the scope of patent application, wherein the generation further includes the first and second scene sets of the maximum value of the data representing f, and is increased by a pre-value within the range of the first and second background sets. Value. 14. The method of item No. in the patent claim, wherein a first set of one or more and a set of moon shellfish materials are generated, including within the range of the first image data set representing the background, and a number representing the smallest data value is generated. A background set and generating a second set or sets of background data sets may be included within the range of the second image data set representing the background to generate a second background set representing the smallest data value. The method called item 14 of the patent scope, wherein the generation further includes the first and second background sets of the minimum value of the background data of the second generation, which are increased within a range of the first and second background sets by a predetermined threshold. Value. 16. The method according to the Chinese Patent Application, wherein a first set of background data is generated, including sampling of the first image data I, and a second set of background data collection includes sampling of the second image data set. Π. For example, the method of item 16 of the patent scope is requested, in which the first group—or more .. and the U member data set 'may include multiple sampling of image data sets maintained within the scope of each background data set, And generating the first group—or multiple groups of transcript collections may include multiple sampling of the second image data set maintained within the range of each background data set. 18 • If the method of item 17 of the patent scope is requested, wherein each first background data set is generated, it may include selecting from multiple sampling to represent the first image data at noon h Η Each patent within the scope of patent application. The data set can include self-consciousness ... and generate a single value for each background in the second image data set. The method of range item 18, wherein the selection includes selecting an intermediate value from all the sampled values in each background? ΗΓ set. ::: The method of item 16 of the patent scope, in which the sampling is performed at the scheduled time 21 4 by __, and each sampling may include background-independent data. • The method of item 1 of the scope of the two I1! Patents, wherein the generation includes a comparison of the second set of the second-f set and the background data set, and a comparison of the second set of the second shadow set and the background data set . 22. The method according to item 1 of the scope of patent application, in which a -difference map is generated, which includes listing each unit in the first image data set as two states # and generating a second difference map further includes Each table in the dataset is one of two states, and the two states indicate whether the value is compatible with the background. • If applied for μ patent scope! Method, wherein the detection includes identification of clusters in the second and second differences κ, where each cluster has a unit that looks like you and is within the scope of its related difference map, indicating a unit that is incompatible with the background . 24. If the method according to item 23 of the patent is requested, the identification cluster in the middle includes reducing the difference map to a row, which is calculated by calculating the units in the row range that is incompatible with the background. 25. The method of claim 24, wherein identifying the cluster further includes identifying the rows within the cluster, and categorizing neighboring rows within the cluster. -4 This paper size applies to China National Standard (CNS) Α4 size (210 X 297 male 543323 6. If the method of applying for the scope of the patent No. 25, which also includes identifying the middle line. "" In the cluster scope 27. If the method of applying for the scope of the patent No. 23, to identify the position related to the cluster. Μ M clusters also include identification 28. If the method of applying for the scope of the patent No. π, the position 罾 ~ red,- / 、 The information related to the cluster including the weighted average number of units in the cluster. 29. If the method in the scope of the patent application No. 23, the set is the target object. The detection in "" also includes clustering 30. · If the category f ~ category cluster in the 29th scope of the patent application includes the "inside the cluster (unit, and only the cluster is counted beyond the predetermined critical classification cluster as the target. 31 .: The method of the 29th scope of the patent application, The clustering also includes counting the cells that are different in the cluster and calculating the total number of cells that have been classified as different from the background in the difference map. Only when the ratio of the number of cells in the cluster to the total number of cells exceeds a predetermined threshold The cluster is classified as the target object. 2. The method of claim 23 of the patent scope, wherein the detection step further includes the identification of the secondary cluster within the cluster, which is the indicator end of the target object and the target. Physical location Identification. Among them, the target is the user's 33. If the method of applying for the scope of the first item of the patent is applied. 34. As for the method of applying for the scope of the first item of the patent, it also includes using the absolute position of the target to control the application. The method of the first item of the scope further includes: obtaining the third image and the fourth image of the target object representing different viewpoints; 5- This paper size applies the Chinese National Standard (CNS) A4 specification (210X297 mm) Sixth, the scope of the patent application is to process the third image into the third image_ the fourth image data set; and the fourth image department to the third image data set related to the fourth background background data set; like Beco k processing 'produces the difference between the past and the second image data set, three difference maps, and the fourth difference map. The fourth difference map is generated. "The difference between the data set and the background data set The third relative position of the target in the difference map, and the target in the fourth difference "! The quiet position of the fourth relative position of the object; and the absolute position of the target object (the first, second, third, and fourth relative position objects. &Amp; &amp; w 3 Method, in which the target object is 37. For example, the application of the 35th scholar, woman, and soil a in the scope of patent application, including the use of the target object to control the application. S 38. —kind A method for controlling the pursuit of a target by using a user as a toilet, a smart phone, an intelligible interface, and an interface, the method includes: acquiring images from at least two viewpoints; processing the acquired images to generate an image data set of each acquired image; Compare each image data set with one or more sets of background data sets to generate a difference map for each acquired image; detect the relative position of the target in each difference map; generate the absolute position of the target from the relative position of the target Position; and use absolute position to allow users to interact with computer applications. 543323 B8 ------—___ C8; 'two' one — ^-^ — 39 · If the method of the 38th scope of the patent application, Including: Furnace head 'things <Absolute Set the screen to the screen base related to the computer application; and // Use the mapping position as the interface to the computer application. The method of item 38 of the patent scope includes: by analyzing the change of the absolute position of the target Organizing gestures related to the target; and merging orbiting positions and gestures into an interface with a computer application. 41. A multiple camera tracking system that interfaces with applications running on a computer. The tracking system includes: One or more video cameras configured to provide a variety of viewpoints as desired, and can be operated to generate a series of video images; processing, it can operate to receive the series of video images, and detect objects appearing in the target area The processor can perform the following processing: generate background data sets from video images; generate image data sets for each received video image; compare each image data set with background data sets to generate a difference map for each video set; Λ detection Measure the relative position of the target in each difference map; and generate the absolute position of the target from the relative position of the target, and The absolute position maps to the position indicator connected to the application. 42. For example, the multiple camera tracking system for item 41 of the patent application, where the target can be a human hand. 43 · For the multiple camera tracking system for item 41 of the patent application, where the target This paper size applies to China National Standard (CNS) A4 (210 X 297 mm) 543323 D2. 4. 1 5 The target area is set in front of the photographic display connected to the computer, and the processing state is further operable to map the absolute position of the target to the position indicator, so that the position indicator position on the photographic display is aligned with the target. 44. If the multi-camera tracking system of item 41 of the patent application scope, where the target area is 4; walking at any distance in front of the photographic display connected to the computer, and where the processor is further operable to map the absolute position of the target to The position index is such that the position of the position index on the photographic display is aligned with the position pointed by the target. 45. For example, the multiple camera tracking system for patent application No. 4 丨, wherein the target area can be set at any distance in front of the photographic display connected to the computer, and the processor is operable to map the absolute position of the target to the position indicator 'Make the moving scale of the target consistent with the large displacement of the position indicator on the photographic display. 46. For example, the multiple camera tracking system of item 41 of the patent application scope, wherein the processing unit can be configured to simulate a computer mouse function. 47. For example, the multiple camera tracking system of item 41 of the patent application scope, wherein the processor can be further configured to simulate a computer mouse control button using gestures estimated from the movement of the target. 48. The multiple camera tracking system of item 41 in the scope of patent application, in which the position of a tolerable target within a predetermined period can be triggered by a selection action in the application. 49. The multi-camera tracking system according to item 41 of the patent application, wherein the processor may be further configured to resemble a computer mouse control button based on a target position tolerated within a predetermined period. -8-This paper size applies Chinese National Standard (CNS) a4 specification (21〇x 297mm) 543323 A8 B8 ____ C8 I --------- D8 VI. Application scope of patent 50. If applying for patent The multiple camera tracking system of the range item 41, wherein within a predetermined period, the tolerable target level within the boundary of the interaction display area can be triggered by a selection action in the application. 51. If requested, the multi-camera tracking system of item 41 of the patent scope, wherein the processor can be configured to control the keys with the computer mouse based on the position tolerance within the interactive display area boundary within a predetermined period of time. similar. Liaoning: The multiple camera tracking system for patent application No. 41, wherein the background data set includes data points represented by at least part of a static structure. 53. The multiple camera tracking system of claim 52, wherein at least part of the static structure includes a patterned surface visible to the camera. 54. The multiple camera tracking system of claim 52, wherein the static structure is a window frame. 55. The multiple camera tracking system of claim 52, wherein the static structure includes a string of light. 56. A multi-camera tracking system linked to an application running on a computer, the tracking system includes: two or more video cameras configured to provide various views as desired, and can be operated to generate A series of video images; a processor operable to receive the series of video images and detect objects appearing in the target area, the processor may perform the following processing: generate a background data set from the video images; Generate video data sets for video images; compare each video data set with background data sets, and generate data for each image. 9- This paper is a national standard (⑽) A4 specification (2igχ tearing board) 543323 6. Scope of patent application The difference map of the material set; 7. The relative position of the target in each difference map; 2 The relative position of the target produces the absolute position of the target; Identify the sub-region in the target area; 7 Identify the sub-region occupied by the target ; When the target object occupies the identified sub-area, the action related to the sub-area is started; and mouth 0 interacts this action with the application. 57. The multi-camera tracking system of item 56 of the patent scope can be a human hand. The multi-camera tracking system of item 56 in the eighth scope, which can be simulated and linked with the application. 59. For example, the multi-camera tracking system of item 56 in the patent application range,-in any sub-region within a predetermined period Inner stick: triggered by action. ^ Location of object and opposite -10- This paper size applies Chinese National Standard (CNS) A4 (210X 297 mm)