CN101479782B - Multi-Input Game Control Mixer - Google Patents

Abstract

Systems and methods of analyzing game control input data are disclosed. Machine-readable media containing instructions for analyzing game control input data are also disclosed.

Description

Multi-Input Game Control Mixer

claim priority

This application claims the following priority: U.S. Patent Application No.11/381,729, Xiao Dong Mao, the invention name is ULTRA SMALL MICROPHONE ARRAY, (Agent No. SCEA05062US00), the filing date is May 4, 2006; U.S. Patent Application No. No.11/381,728, Xiao Dong Mao, the name of the invention is ECHO AND NOISE CANCELLATION, (Agent No. SCEA05064US00), the filing date is May 4, 2006; U.S. Patent Application No.11/381,725, Xiao Dong Mao, the name of the invention It is "METHODS AND APPARATUS FOR TARGETEDSOUND DETECTION", (Agent No. SCEA05072US00), the application date is May 4, 2006; U.S. Patent Application No. 11/381,727, Xiao Dong Mao, the invention name is "NOISEREMOVAL FOR ELECTRONIC DEVICE WITH FAR FIELD MICROPHONEON CONSOLE", (Agent No. SCEA05073US00), the application date is May 4, 2006; U.S. Patent Application No. 11/381,724, Xiao Dong Mao, the invention name is "METHODS ANDAPPARATUS FOR TARGETED SOUND DETECTION ANDCHARACTERIZATION", (Agent No. SCEA05079US00), the application date is May 4, 2006; U.S. patent application No. 11/381,721, Xiao Dong Mao, the invention name is "SELECTIVESOUND SOURCE LISTENING IN CONJUCTION WITH COMPUTERINTERACTIVE PROCESSING", (Agent No. SCEA04005US00JUMBOUS), the application date for May 4, 2006. The above documents are incorporated herein by reference.

This application also claims the following priority: U.S. Patent Application No. 11/382,031, Gary Zalewski et al., titled "MULTI-INPUT GAME CONTROL MIXER", (Attorney No. SCEA06MXR1), filed May 6, 2006 ; U.S. Patent Application No.11/382,032, Gary Zalewski, etc., the invention name is "SYSTEM FOR TRACKING USERMANIPULATIONS WITHIN AN ENVIRONMENT", (Agent No. SCEA06MXR2), and the application date is May 6, 2006. The above documents are incorporated herein by reference.

This application also claims priority to the following co-pending patent application: Application No. 11/418,988, Xiao Dong Mao, entitled "METHODS AND APPARATUSES FORADJUSTING A LISTENING AREA FOR CAPUTURING SOUNDS", (Attorney No. SCEA-00300 ), filed on May 4, 2006, the entire contents of which are incorporated herein by reference. This application also claims priority to the following co-pending patent application: Application No. 11/418,989, Xiao Dong Mao, entitled "METHODS AND APPARATUSES FORCAPTURING AN AUDIO SIGNAL BASED ON VISUAL IMAGE", (Attorney No. SCEA- 00400), filed May 4, 2006, the entire contents of which are incorporated herein by reference. This application also claims priority to the following co-pending patent application: Application No. 11/429,047, Xiao Dong Mao, titled "METHODS AND APPARATUSES FORCAPTURING AN AUDIO SIGNAL BASED ON A LOCATION OF THESIGNAL", (Attorney No. SCEA-00500), filed May 4, 2006, the entire contents of which are incorporated herein by reference. This application also claims priority to the following co-pending patent application: Application No. 11/429,133, Richard Marks et al., entitled "SELECTIVE SOUNDSOURCE LISTENING IN CONJUNCTION WITH COMPUTER INTERACTIVE PROCESSING", (Attorney No. SCEA04005US01-SONYP045), The filing date is May 4, 2006, which is hereby incorporated by reference in its entirety. This application also claims priority to the following co-pending patent application: Application No. 11/429,414, Richard Marks et al., entitled "Computer Image and Audio Processing of Intensity and Input Devices for Interfacing With A Computer Program", (Attorney No. SONYP052), the filing date is May 4, 2006, the entire content of which is incorporated herein by reference.

This application also claims priority to the following co-pending patent application: Application No. 11/382,033, titled "SYSTEM, METHOD, AND APPARATUS FORTHREE-DIMENSIONAL INPUT CONTROL", (Attorney No. SCEA06INRT1), filed on Incorporated by reference in its entirety on May 6, 2006. This application also claims priority to the following co-pending patent application: Application No. 11/382,035, titled "INERTIALLY TRACKABLE HAND-HELD CONTROLLER", (Attorney No. SCEA06INRT2), filed May 6, 2006 date, the entire contents of which are incorporated herein by reference. This application also claims the priority of the following co-pending patent application: Application No. 11/382,036, the title of the invention is "METHODAND SYSTEM FOR APPLYING GEARING EFFECTS TO VISUAL TRACKING", (Agent No. SONYP058A), the filing date is May 2006 The entire contents of which are incorporated herein by reference. This application also claims the priority of the following co-pending patent application: the application number is 11/382,041, the title of the invention is "METHOD AND SYSTEM FOR APPLYING GEARING EFFECTS TO INERTIAL TRACKING", (the agency number is SONYP058B), and the application date is 2006 Incorporated by reference in its entirety on May 6, 2009. This application also claims the priority of the following co-pending patent application: Application No. 11/382,038, the title of the invention is "METHOD AND SYSTEM FOR APPLYINGGEARING EFFECTS TO ACOUSTICAL TRACKING", (Agent No. SONYP058C), the filing date is 2006 Incorporated by reference in its entirety on May 6. This application also claims priority to the following co-pending patent application: Application No. 11/382,040, titled "METHOD AND SYSTEM FOR APPLYING GEARING EFFECTSTO MULTI-CHANNEL MIXED INPUT", (Attorney No. SONYP058D), dated As of May 6, 2006, the entire contents of which are incorporated herein by reference. This application also claims priority to the following co-pending patent application: Application No. 11/382,034, titled "SCHEMEFOR DETECTING AND TRACKING USER MANIPULATION OF A GAMECONTROLLER BODY", (Attorney No. SCEA05082US00), filed 2006 Incorporated by reference in its entirety on May 6, 2009. This application also claims priority to the following co-pending patent application: Application No. 11/382,037, titled "SCHEME FORTRANSLATING MOVEMENTS OF A HAND-HELD CONTROLLER INTOINUTS FOR A SYSTEM", (Attorney No. 86324), application dated May 6, 2006, which is hereby incorporated by reference in its entirety. This application also claims priority to the following co-pending patent application: Application No. 11/382,043, titled "DETECTABLE ANDTRACKABLE HAND-HELD CONTROLLER", (Attorney No. 86325), filed May 6, 2006 date, the entire contents of which are incorporated herein by reference. This application also claims priority to the following co-pending patent application: Application No. 11/382,039, titled "METHODFOR MAPPING MOVEMENTS OF A HAND-HELD CONTROLLER TO GAME COMMANDS", (Attorney No. 86326), filed on Incorporated by reference in its entirety on May 6, 2006. This application also claims priority to the following co-pending patent application: Application No. 29/259,349, titled "CONTROLLER WITH INFRAREDPORT ((DESIGN PATENT))", (Attorney No. SCEA06007US00), filed 2006 Incorporated by reference in its entirety on May 6. This application also claims priority to the following co-pending patent application: Application No. 29/259,350, titled "CONTROLLER WITH TRACKING SENSORS ((DESIGN PATENT))", (Attorney No. SCEA06008US00), filed 2006 Incorporated by reference in its entirety on May 6. This application also claims priority to the following co-pending patent application: Application No. 60/798,031, titled "DYNAMIC TARGET INTERFACE", (Attorney No. SCEA06009US00), filed May 6, 2006, by Its entire content is incorporated herein by reference. This application also claims priority to the following co-pending patent application: Application No. 29/259,348, titled "TRACKED CONTROLLER DEVICE ((DESIGN))", (Attorney No. SCEA06010US00), filed May 2006 The entire contents of which are incorporated herein by reference.

Cross-Referenced Related Applications

This application relates to U.S. Patent Application No. 10/207,677, the name of the invention is "MAN-MACHINE INTERFACE USING A DEFORMABLE DEVICE", and the application date is July 27, 2002; U.S. Patent Application No. 10/650,409, the name of the invention is " AUDIO INPUT SYSTEM", the filing date is August 27, 2003; US Patent Application No. 10/663,236, the invention name is "METHODAND APPARATUS FOR ADJUSTING A VIEW OF A SCENE BEING DISPLAYED ACCORDING TO TRACKED HEAD MOTION", the filing date is 2003 September 15, 2004; U.S. Patent Application No. 10/759,782, the name of the invention is "METHOD ANDAPPARATUS FOR LIGHT INPUT DEVICE", the application date is January 16, 2004; U.S. Patent Application No. 10/820,469, the name of the invention It is "METHOD AND APPARATUS TODETECT AND REMOVE AUDIO DISTURBANCES", the application date is April 7, 2004; US Patent Application No. 11/301,673, the title of the invention is "METHOD FOR USINGRELATIVE HEAD AND HAND POSITIONS TO ENABLE A POINTINGINTERFACE VIACAMERATRACKING", The application date is December 12, 2005; U.S. provisional patent application No. 60/718,145, the invention name is "AUDIO, VIDEO, SIMULATION, AND USER INTERFACE PARADIGMS", the application date is September 15, 2005, by reference The entire content of which is incorporated herein.

technical field of invention

The present invention relates generally to human-machine interfaces, and more particularly to processing multi-channel input to track user manipulation of one or more controllers.

Background Art of the Invention

Computer entertainment systems often include handheld controllers, game controllers, or other controllers. A user or player uses the controller to send commands or other instructions to the entertainment system to control a video game or other simulation being played. For example, the controller may be provided with a user-operated manipulator, such as a joystick. The manipulated variable of the joystick is converted from an analog value to a digital value, and the digital value is sent to the main unit of the game machine. The controller may also be provided with buttons that the user can operate.

The present invention is a further development of the above or other background information elements.

Detailed description of the drawings

The teachings of the present invention can be readily understood by considering the following detailed description when taken in conjunction with the accompanying drawings, in which:

Figure 1 is a physical circuit diagram illustrating a video game system operating in accordance with one embodiment of the present invention;

Figure 2 is a perspective view of a controller made in accordance with one embodiment of the present invention;

Figure 3 is a three-dimensional schematic diagram illustrating an accelerometer that may be used in a controller according to one embodiment of the present invention;

Figure 4 is a block diagram of a system for mixing multiple control inputs according to one embodiment of the present invention;

5A is a block diagram of a portion of the video game system of FIG. 1;

5B is a flowchart of a method for tracking a controller of a video game system according to one embodiment of the present invention;

5C is a flowchart of a method of utilizing position and/or orientation information during a game played in a video game system according to one embodiment of the present invention;

Figure 6 is a block diagram of a video game system according to one embodiment of the present invention; and

7 is a block diagram of a microprocessor actuator of a video game system according to one embodiment of the present invention.

Detailed ways

Although the following detailed description contains many specific details for purposes of illustration, any person of ordinary skill in the art will appreciate that variations or modifications to the following details are within the scope of the invention. Thus, the exemplary embodiments of the present invention described below are illustrated without lack of generality and without affecting the scope of the claimed invention.

Embodiments of the various methods, apparatus, schemes, and systems described herein provide for detection, capture, and tracking of user movements, actions, and/or manipulations of the entire controller body itself. Detected user movements, actions and/or manipulations of the entire controller body may serve as additional commands to control various aspects of an ongoing game or other simulation.

Detecting and tracking user manipulation of the game controller body can be accomplished in a number of ways. For example, inertial sensors, such as accelerometers or gyroscopes, and image capture units, such as digital cameras, can be used in computer entertainment systems to detect movements of the handheld controller body and translate them into in-game actions. Examples of tracking controllers with inertial sensors have been described, for example, in U.S. Patent Application Serial No. 11/382,033, entitled "SYSTEM, METHOD, AND APPARATUS FORTHREE-DIMENSIONNAL INPUT CONTROL" (Attorney No. SCEA06INRT1), which is incorporated by reference way incorporated into this article. Examples of tracking a controller with an image capturer have been described, for example, in U.S. Patent Application No. 11/,382,034, entitled "SCHEME FORDETECTING AND TRACKING USER MANIPULATION OF A GAME CONTROLLER BODY" (Attorney No. SCEA05082US00), which was adopted by incorporated herein by reference. Additionally, the controller and/or user may also be tracked acoustically through the use of a microphone array or appropriate signal processing. An example of such acoustic tracking has been described in US Patent Application No. 11/381,721, which is incorporated herein by reference.

Acoustic sensors, inertial sensors and image captures can be used alone or in any combination to detect many different types of motion of the controller, such as up and down movement, twist movement, side to side movement, jerk movement, stick Type of movement, sudden drop movement and so on. These actions may correspond to various commands such that the aforementioned actions are translated into in-game actions. Detecting and tracking the user's manipulation of the game controller body can be applied to implement many different types of games, simulations, etc., allowing the user, for example, to engage in fencing or lightsaber combat, to trace the shape of an item with a stick, to participate in many different types of sporting events, participate in on-screen battles or other encounters, and more. The game program can be configured to track the motion of the controller and recognize certain pre-recorded gestures from the tracked motion. Recognition of one or more of the aforementioned gestures can trigger a game state change.

In an embodiment of the present invention, controller path information obtained from the different sources described above may be blended prior to analysis for gesture recognition. Tracking data from different sources (eg, acoustic, inertial, and image capture) can be blended in a way that increases the likelihood of gesture recognition.

Referring to FIG. 1 , there is shown a system 100 operating in accordance with one embodiment of the present invention. As shown, a computer entertainment console 102 may be coupled to a television or other video display 104 for displaying images of a video game or other simulation. The game or other simulation may be stored on a DVD, CD, flash memory, USB memory or other storage medium 106 inserted into the console 102 . A user or player 108 manipulates a game controller 110 to control a video game or other simulation. As shown in FIG. 2 , the game controller 110 includes an inertial sensor 112 that generates a signal in response to the position, motion, direction or change in direction of the game controller 110 . In addition to inertial sensors, game controller 110 may include conventional control input devices such as joystick 111, buttons 113, R1, L1, and the like.

During operation, user 108 physically moves controller 110 . For example, the controller 110 may be moved by the user 108 in any direction, such as up, down, side to side, side to side, twisting, rolling, shaking, jerking, dipping, and the like. The aforementioned movement of the controller 110 itself may be detected and captured by the camera 114, which is tracked by analyzing signals from the inertial sensors 112 as described below.

Referring again to FIG. 1 , the system 100 optionally includes a video camera or other video image capture device 114 that may be positioned such that the controller 110 is within the field of view 116 of the camera. Analysis of images from image capture device 114 may be used in conjunction with analysis of data from inertial sensors 112 . As shown in FIG. 2, the controller 110 may optionally be configured with light sources, such as light emitting diodes (LEDs) 202, 204, 206, 208, to facilitate tracking by video analysis. The above-mentioned means may be assembled into the body of the controller 110 . Here, the term "body" is meant to describe the handheld portion (or worn portion, if the game controller is wearable) of the game controller 110 .

Analysis of such video images for tracking controller 110 has been disclosed, for example, in U.S. Patent Application No. 11/382,034, titled "SCHEME FOR DETECTING AND TRACKINGUSER MANIPULATION OF A GAME CONTROLLER BODY" (Attorney No. SCEA05082US00), The aforementioned documents are incorporated herein by reference. Console 102 may include an acoustic sensor, such as microphone array 118 . Controller 110 may also include an audio signal generator 210 (e.g., a speaker) to provide an audio source to facilitate audio tracking of controller 110 with microphone array 118 and appropriate audio signal processing, e.g., in U.S. Patent Application No. 11 /381,724, which is hereby incorporated by reference.

Typically, signals from inertial sensors 112 are used to generate position and orientation data for controller 110 . The above data can be used to calculate many physical aspects of the movement of the controller 110, such as the acceleration and velocity of the controller along any axis, the tilt, pitch, yaw, roll of the controller, and any telemetry points of the controller 110. Here, telemetry generally refers to measuring and reporting systems over long distances or information of interest to the designer or operator of the system.

The ability to detect and track movements of the controller 110 enables a determination of whether the controller 110 has performed any predefined movements. That is, specific movement patterns or gestures of the controller 110 can be pre-defined and can be used as input commands for games or other simulations. For example, a jumping gesture of the controller 110 may be defined as a command, a twisting gesture of the controller 110 may be defined as another command, a shaking gesture of the controller 110 may be defined as another command, and so on. In this way, the way the user 108 physically moves the controller 110 is used as another input to control the game, providing the user with a more exciting and fun experience.

By way of example and not limitation, inertial sensor 112 may be an accelerometer. Figure 3 illustrates an example of an accelerometer 300 in the form of a simple mass 302 elastically connected to a frame 304 at four points, for example via springs 306, 308, 310, 312. Pitch and roll axes (represented by X and Y axes, respectively) lie in planes transverse to the frame. A yaw (yaw) axis Z is in a direction perpendicular to a plane containing a pitch axis X and a roll axis Y. Frame 304 is mounted on controller 110 in any suitable manner. As the frame 304 (and joystick controller 110) accelerates and/or rotates, the mass 302 can be displaced relative to the frame 304, and the springs 306, 308, 310, 312 can depend on the amount and direction of the translational and/or rotational acceleration and and/or pitch and/or roll and/or yaw angle elongation or compression. The displacement of the block 302 and/or the extension or compression of the springs 306, 308, 310, 312 can be sensed, for example, using appropriate sensors 314, 316, 318, 320, and then converted to a known or determinable The way the acceleration signal depends on the amount of pitch and/or roll.

There are a number of different ways to track the position of the block and/or the force applied to it, including resistive strain gauge materials, optical sensors, magnetic sensors, Hall effect devices, piezoelectric devices, capacitive sensors, and the like. Embodiments of the invention may include any number and type or combination of types of sensors. By way of example, but not limitation, sensors 314 , 316 , 318 , 320 may be gap closing electrodes placed above block 302 . As the position of the block changes relative to each electrode, the capacitance between the block and each electrode changes. Each electrode may be connected to a circuit that generates a signal about the capacitance of the block 302 relative to that electrode (and thus about the proximity of the block 302 to that electrode). Additionally, the springs 306, 308, 310, 312 may include resistive strain gauge sensors that generate signals regarding spring compression or elongation.

In some embodiments, frame 304 is gimbal mounted to controller 110 such that accelerometer 300 maintains a fixed orientation about the pitch and/or roll and/or yaw axes. In the manner described above, the controller axes X, Y, Z can be directly mapped to the corresponding axes in the real space without having to consider the inclination of these controller axes with respect to the coordinate axes of the real space.

As described above, data from inertial, image capture, and acoustic sources may be analyzed to generate a path that tracks the position and/or orientation of the controller 110 . As shown in the block diagram of FIG. 4 , a system 400 according to one embodiment of the present invention includes an inertial analyzer 402 , an image analyzer 404 and an acoustic analyzer 406 . Each of the analyzers described above receives a signal from the sensed environment 401 . The analyzers 402, 404, 406 can be implemented by hardware, software (or firmware) or some combination of the above two or more. Each analyzer produces tracking information about the position and/or orientation of an object of interest. As an example, the object of interest may be the controller 110 described above. Image analyzer 404 may operate in conjunction with the methods disclosed in US Patent Application Serial No. 11/382,034 (Attorney No. SCEA05082US00), forming the following field regarding these methods. Inertial analyzer 402 can operate in conjunction with the methods disclosed in U.S. Patent Application No. 11/382,033, titled "SYSTEM, METHOD, AND APPARATUS FOR THREE-DIMENSIONAL INPUT CONTROL" (Attorney No. SCEA06INRT1), forming the following fields about these methods . Acoustic analyzer 406 may operate using the methods disclosed in US Patent Application No. 11/381,724, forming the following fields regarding these methods.

Analyzers 402, 404 or 406 may be considered to be associated with different input channels of position and/or orientation information. Mixer 408 may receive multiple input channels, such channels may contain sampled data describing sensed environment 401, typically observations from that channel. The position and/or orientation information generated by inertial analyzer 402 , image analyzer 404 and acoustic analyzer 406 may be coupled into an input of mixer 408 . Mixer 408 and analyzers 402, 404, 406 may be queried by gaming software program 410 and may be configured to interrupt gaming software in response to events. Events may include gesture recognition events, gearing changes, configuration changes, setting noise levels, setting sample rates, changing mapping chains, etc., examples of which are discussed below. Mixer 408 may operate in conjunction with the methods described herein and form the following fields regarding these methods.

As mentioned above, signals from different input channels, such as inertial sensors, video imagery and/or acoustic sensors, can be analyzed by inertial analyzer 402, image analyzer 404 and acoustic analyzer 406, respectively, to play video games according to the inventive method. The action and/or direction of the controller 110 is determined during the process. The methods described above can be implemented as a series of processor-executable program code instructions stored on a processor-readable medium and executed on a digital processor. For example, as shown in FIG. 5A , a video game system 100 may be included on a console 102 having an inertial analyzer 402 , an image analyzer 404 and an acoustic analyzer 406 implemented either in hardware or in software. As an example, analyzers 402 , 404 , 406 may be implemented as software instructions running on a suitable processor unit 502 . As an example, processor unit 502 may be a digital processor, such as a microprocessor of the type commonly used in video game consoles. A portion of the instructions may be stored in memory 506 . Optionally, the inertial analyzer 402, the image analyzer 404 and the acoustic analyzer 406 may be implemented by hardware, such as application-specific integrated circuits (ASICs). Such analyzer hardware may be located on controller 110 or console 102, or may be located remotely elsewhere. In a hardware implementation, the analyzers 402, 404, 406 may be programmable in response to external signals, e.g., from the processor 502, or other remotely located sources, e.g. connected on.

Inertial analyzer 402 may include or execute instructions for analyzing signals generated by inertial sensors 112 and utilizing information regarding the position and/or orientation of controller 110 . Likewise, the image analyzer 404 can execute instructions for analyzing the images captured by the image capture unit 114 . Additionally, the acoustic analyzer may execute instructions to analyze images captured by the microphone array 118 . As shown in flowchart 510 of FIG. 5B , the aforementioned signals and/or images may be received by analyzers 402 , 404 , 406 , as shown in block 512 . The aforementioned signals and/or images may be analyzed by analyzers 402, 404, 406 to determine inertial tracking information 403, image tracking information 405, and acoustic tracking information 407 regarding the position and/or orientation of controller 110, as shown in block 514 of. Tracking information 403, 405, 407 may relate to one or more degrees of freedom. Preferably six degrees of freedom are tracked to describe the manipulation of the controller 110 or other tracked object. Such degrees of freedom may involve controller pitch, yaw, roll and position, velocity or acceleration along the x, y and z-axes.

Mixer 408 mixes inertial information 403 , image information 405 and acoustic information 407 to produce improved position and/or orientation information 409 as indicated by block 516 . As an example, the mixer 408 may apply different weights to the inertial, image and acoustic tracking information 403, 405, 407 based on game or environmental conditions and then take a weighted average. Additionally, mixer 408 may contain its own hybrid analyzer 412 for analyzing the combined position/orientation information and producing its own resulting "mixer" information that is combined with information produced by other analyzers.

In one embodiment of the invention, mixer 408 may assign distribution values to trace information 403 , 405 , 407 from analyzers 402 , 404 , 406 . As mentioned above, certain sets of input control data can be averaged. In this embodiment, however, input control data is assigned a value prior to averaging, whereby input control data from some analyzers is more analytically important than from others.

Mixer 408 can employ many functions in the context of the current system, including observing, correcting, stabilizing, deriving, combining, routing, blending, reporting, buffering, interrupting other processing and analysis. The functions described above may be performed with respect to trace information 403 , 405 , 407 received from one or more analyzers 402 , 404 , 406 . Each analyzer 402, 404, 406 can receive and/or derive specific trace information, and the mixer 408 can be implemented to optimize the use of the received trace information 403, 405, 407 and produce improved trace information 409.

Analyzers 402, 404, 406 and mixer 408 are preferably configured to provide trace information in a similar output format. A trace information parameter from any analyzer element 402, 404, 406 can be mapped to a single parameter in the analyzer. Optionally, mixer 408 may form trace information for any analyzer 402 , 404 , 406 by processing one or more trace information parameters from one or more analyzers 402 , 404 , 406 . A mixer may combine two or more elements of traces of the same parameter type obtained from analyzers 402, 404, 406 and/or perform functions across multiple parameters of traces produced by these analyzers to create A comprehensive output set that has the benefit of being generated from multiple input channels.

Improved tracking information 409 may be applied during video game play with system 100 , as shown at block 518 . In some embodiments, position and/or orientation information may be used during game play in relation to gestures made by the user 108 . In some embodiments, mixer 408 may operate with gesture recognizer 505 to associate at least one action in the gaming environment with one or more user actions from the user (e.g., manipulation of a controller in space) .

As shown in flowchart 520 in FIG. 5C , the path of controller 110 may be tracked using position and/or orientation information, as shown in block 522 . By way of example, but not limitation, a path may include a set of points representing the position of the controller's center of mass with respect to some coordinate system. Each location point can be represented by one or more coordinates, such as X, Y and Z coordinates in a Cartesian coordinate system. Time can be associated with each point on the path so that the shape of the path and the progress of the controller along the path can be monitored. Additionally, each point in the set of points may be associated with data representing the orientation of the controller, eg, one or more rotation angles of the controller about its center of mass. Additionally, each point on the path may be associated with values of the velocity and acceleration of the controller's center of mass, as well as the rate and angular acceleration of the controller's angular rotation about its center of mass.

As indicated at block 524, the tracked path may be compared to one or more stored paths corresponding to known and/or pre-recorded gestures 508 associated with the context of the ongoing video game. Recognizer 505 may be configured to recognize a user or process audio authentication gestures, among other things. For example, a user may be recognized by recognizer 505 through a gesture, and the gesture may be specific to the user. The particular gestures described above may be recorded and included in pre-recorded gestures 508 stored in memory 506 . The recording process can optionally store the sounds produced during the attitude recording process. The sensed environment is sampled to a multi-channel analyzer and processed. The processor may refer to the attitude model to determine and identify and/or identify a user or target based on sound or acoustic patterns with a high degree of accuracy and performance.

As shown in FIG. 5A , data 508 representative of gestures may be stored in memory 506 . Examples of gestures include, but are not limited to, throwing a target such as a ball, swinging a target such as a club or golf club, pumping a hand pump, opening or closing a door or window, turning a steering wheel or other vehicle control, martial arts actions such as Punch, sanding move, wax on wax off, paint house, shake, rattle, roll, football pitch, knob move, 3D MOUSE move, roll move, known Attitude movement, any recordable movement, moving back and forth along any vector, i.e. pumping tires but in any direction in space, movement along a path, movement with precise stop and start times, in intrinsic noise Any time-based user manipulation within levels, splines, etc. can be recorded, tracked and repeated. Each of these poses can be pre-recorded from path data and stored as a time-based model. The comparison of the path to the stored pose can start with the assumption of a steady state, and if the path deviates from the steady state, the path can be compared to the stored pose through a process of elimination. If there is no match at block 526 , the analyzer may continue to trace the path of the controller 110 at block 522 . If there is a sufficient match between the path (or a portion of the path) and the stored pose, the game state may be changed, as indicated by block 528 . Game state changes may include, but are not limited to, interrupting, sending control signals, changing variables, and the like.

Here is an example of what might happen. Path analyzer 402, 404, 406, or 412 tracks the movement of controller 110 upon determining that controller 110 has left a steady state. These poses are possible "hits" as long as the path of the controller 110 is consistent with the path defined in the stored pose model 508 . If the path of the controller 110 deviates (within the noise tolerance setting) from any pose model 508, that pose model is removed from the hit list. Each pose reference model includes a temporal basis in which the pose is recorded. The analyzer 402, 404, 406, or 412 compares the controller path data with the stored pose 508 at the appropriate time index point. Occurrence of a steady-state condition resets the clock. When deviating from steady state (ie, when movement is tracked outside the noise threshold), the hit list is added to all potential pose models. The clock is started and the controller's moves are compared to the hit list. Again, comparison is a progression through time. If any pose in the hit list reaches the end of that pose, then it's a hit.

In some embodiments, the mixer 408 and/or the individual analyzers 402, 404, 406, 412 may notify the game program when certain events occur. Examples of such events include the following:

Reached zero-acceleration point interrupt (X and/or Y and/or Z axis). In certain game situations, when the acceleration of the controller changes at an inflection point, the analyzer can notify or interrupt the process within the game program. For example, user 108 may use controller 110 to control a game avatar representing a quarterback in a simulated football game. The analyzer may follow the controller (representing a football) through a path generated from the signals from the inertial sensors 112 . A particular change in the acceleration of the controller 110 may indicate the release of the football. At this point, the analyzer may trigger another process in the program (eg, a physics simulation package) to simulate the trajectory of the football based on the position of the controller at the release point, and/or the velocity and/or direction.

New gesture recognition interrupt

Additionally, analyzers can be configured by one or more inputs. Examples of such input include, but are not limited to:

Set the noise level (X, Y or Z axis). The noise level can be a reference tolerance value when analyzing the user's hand shake in the game.

Set the sample rate. As used herein, sampling rate may refer to how often an analyzer samples signals from an inertial sensor. The sample rate can be set to oversample or average the signal.

Set up the transmission. As used herein, gear generally refers to the ratio of controller movement to movement occurring in the game. An example of such "actuation" in the context of video game control can be found in U.S. Patent Application Serial No. 11/382,040, filed May 7, 2006, (Attorney Docket No. SONYP058D), which is incorporated by reference Incorporated into this article.

Set up mapping chains. As used herein, a map chain refers to a map of pose models. Pose model mapping can be done on specific input channels (eg only for path data generated from inertial sensor signals) or on mixed channels formed in the mixer unit. The three input channels may be served by two or more different analyzers like inertial analyzer 402 . In particular, these analyzers may include: Inertial Analyzer 402 as used herein, Video Analysis as described in U.S. Patent Application Serial No. 11/382,034 entitled SCHEMEFOR DETECTING AND TRACKING USER MANIPULATION OF A GAME CONTROLLER BODY (Attorney No. SCEA05082US00) instrument, which is incorporated herein by reference, and the acoustic analyzer described in US Patent Application No. 11/381,721, which is incorporated herein by reference. Analyzers can be configured with mapping chains. Mapchains can be swapped out by the game during game play, as can settings for the analyzer and for the mixer.

Referring again to FIG. 5B , block 512 , those skilled in the art will appreciate that there are a variety of ways to generate signals from inertial sensors 112 . Some examples of these are described here. Referring to block 514 , there are a number of ways to analyze the sensor signals generated in block 512 to obtain tracking information about the position and/or orientation of the controller 110 . By way of example, but not limitation, the tracking information may include information on the following parameters alone or in any combination, but not limited thereto.

Controller orientation. The orientation of the controller 110 may be expressed as a pitch, roll or yaw angle (eg, in radians) relative to some reference orientation. The rate of change of controller orientation (eg, angular velocity or acceleration) may also be included in the position and/or orientation information. For example, where inertial sensors 112 comprise gyroscopic sensors, controller orientation information may be obtained directly in the form of one or more output values proportional to pitch, roll, or yaw angles.

Controller position (e.g., Cartesian coordinates X, Y, Z of controller 110 in some frame of reference)

Controller X-axis speed

Controller Y-axis speed

Controller Z-axis speed

Controller X-axis acceleration

Controller Y-axis acceleration

Controller Z-axis acceleration

It should be noted that with respect to position, velocity and acceleration, position and/or orientation information may be expressed in a coordinate system other than Cartesian. For example, cylindrical or spherical coordinates can be used for position, velocity and acceleration. Acceleration information about the X, Y, Z axes can be obtained directly from accelerometer type sensors, such as described herein. X, Y, Z acceleration can be integrated with respect to time from some starting moment to determine the change in X, Y, Z velocity. These velocities can be calculated by adding the velocity change to the known X-, Y-, Z-velocity values at an initial moment in time. X, Y, Z velocity can be integrated with time to determine the X-, Y-, Z-displacement of the controller. The X-, Y-, Z-positions can be determined by adding displacements to the X-, Y-, Z-positions known at the initial time.

Steady State Y/N - This specific information indicates whether the controller is in a steady state, which can be defined as any position, which is also subject to change. In a preferred embodiment, the steady state position may be a position in which the controller is held in a more or less horizontal orientation at approximately waist level with the user.

Time to last stable state generally refers to data on how long a period of time has elapsed since a stable state was last detected (as described above). As mentioned earlier, the determination of this time can be calculated in real time, processor cycles or sampling cycles. Time-to-Last Steady State Data The time-to-last-steady-state data time may be important for tracking controller resets with respect to the initial point, to ensure accurate mapping of characters or objects in the game environment. This data may also be important (exclusively and inclusively) for determining available actions/gestures that may then be performed in the gaming environment.

The last recognized gesture generally refers to the last gesture recognized by the gesture recognizer 505 (implemented by hardware or software). Confirmation of the last recognized gesture may be important for the fact that previous gestures may be related to possible gestures that are subsequently recognized or other actions that take place in the game environment.

Last gesture recognition time

Each of the above outputs can be sampled at any time by the game program or software.

In an embodiment of the invention, mixer 408 may assign distribution values to trace information 403 , 405 , 407 from analyzers 402 , 404 , 406 . As mentioned above, certain sets of input control data can be averaged. In this embodiment, however, input control data is assigned a value prior to averaging, whereby input control data from some analyzers is more analytically important than from others.

For example, mixer 408 may require tracking information on acceleration and steady state. The mixer 408 will then receive the tracking information 403, 405, 407 as described above. Tracking information may include parameters regarding, for example, acceleration and steady state as described above. The blender 408 may assign distribution values to the tracking information data sets 403, 405, 407 prior to averaging the data representing the above information. For example, the x- and y-acceleration parameters obtained from inertial analyzer 402 may be weighted by a value of 90%. However, the x- and y-acceleration data obtained from image analyzer 406 may only be 10% weighted. Acoustic analyzer trace information 407 pertaining to acceleration parameters can be weighted 0%, that is to say the data has no weight.

Similarly, the Z-axis tracking information parameters obtained from the inertial analyzer 402 may be weighted 10%, while the image analyzer Z-axis tracking information may be weighted 90%. The acoustic analyzer trace 407 may again be weighted with a 0% value, but the steady state trace from the acoustic analyzer 406 may be weighted at 100%, while the other analyzer traces are weighted at 0%.

After the appropriate distribution weights are assigned, the input control data can be averaged with the weights to arrive at a weighted average input control data set which is then analyzed by gesture recognizer 505 and associated with specific actions in the game environment . The associated values may be predefined by mixer 408 or by a particular game title. The above values may also be the result of mixer 408 determining certain characteristics of the data from each analyzer and dynamically adjusting accordingly, as discussed further below. The adjustments described above may also be the result of establishing a historical knowledge base of when certain data had certain values in certain circumstances and/or in response to specificities of a given game title.

Mixer 408 may be configured to operate dynamically during game play. For example, when the mixer 408 receives input control data, it may recognize that certain data has been out of acceptable range or data quality, or reflect bad data which may indicate a processing error at the associated input device.

Also, certain conditions may change in real-world environments. For example, the natural light in a user's home gaming environment may increase from morning to afternoon, causing problems with image data capture. Additionally, neighbors or family members may become rowdy as the day progresses, causing problems with sound data capture. Similarly, if users continue to play for several hours, their reflexes may become less responsive, causing problems in the interpretation of inertial data.

In these examples, or in any other example where the quality of a particular form of input control data is questionable, mixer 408 may dynamically reassign distribution weights for particular data sets from particular devices such that more or less The importance of is assigned to specific input control data as described above. Similarly, the game environment may change during the course of a game where the requirements of a particular game change requiring reassignment of weights or needs for particular input control data.

Similarly, mixer 408 can identify based on processing errors or feedback data that may be generated by gesture recognizer 505 that some data being passed to gesture recognizer 505 is being processed incorrectly, slowly, or not at all. In response to this feedback or identifying these processing difficulties (e.g., an error occurred when gesture recognizer 505 made the association although the image analysis data was within acceptable limits), mixer 408 may adjust which analyzer to use from, and if possible When, which input to seek control data. The mixer 408 may also require specific analysis and processing of the input control data by a suitable analyzer before it is passed to the mixer 408 where the data will be processed again (e.g., averaging the data) to obtain more One layer of assurance that the data passed to gesture recognizer 505 will be efficiently and properly processed.

In some embodiments, mixer 408 can recognize that certain data is bad, invalid, or outside of a particular variable, and can access specific input control data or variables associated with that data, making it replaceable Incorrect data, or specific data on required variables to properly analyze and calculate.

According to an embodiment of the present invention, a video game system and method of the type described above may be implemented as described in FIG. 6 . Video game system 600 may include processor 601 and memory 602 (such as RAM, DRAM, ROM, etc.). Additionally, video game system 600 may have multiple processors 601 if parallel processing is to be performed. Memory 602 includes data and game program code 604, which may include portions configured as described above. In particular, memory 602 may include inertial signal data 606, which may include the stored controller path information described above. Memory 602 may also contain stored gesture data 608 , such as data representing one or more gestures associated with game program 604 . The encoded instructions executing on the processor 602 may implement a multi-input mixer 605 that may be configured and operated as described above.

System 600 may also include well-known support functions 610 such as input/output (I/O) elements 611 , power supplies (P/S) 612 , clocks (CLK) 613 and buffers 614 . Device 600 may optionally include a mass storage device 615, such as a magnetic disk drive, CD-ROM drive, tape drive, etc., to store programs and/or data. The controller may also optionally include a display unit 616 and a user interface unit 618 to facilitate interaction between the controller 600 and the user. The display unit 616 may be in the form of a cathode ray tube (CRT) or a flat screen displaying text, numbers, graphic symbols or images. User interface 618 may include a keyboard, mouse, joystick, light pen, or other device. Additionally, user interface 618 may include a microphone, video camera, or other signal conversion device to provide direct capture of the signal to be analyzed. The processor 601, the memory 602 and other components of the system 600 can exchange signals (such as code instructions and data) with each other through the system bus 620 shown in FIG. 6 .

Microphone array 622 may be coupled to system 600 through I/O function 611 . The microphone array may comprise about 2 to 8 microphones, preferably about 4 microphones, with adjacent microphones separated by a distance of less than about 4 cm, preferably between about 1 and 2 cm. Preferably, the microphones in array 622 are omnidirectional microphones. An optional image capture unit 623 , such as a digital video camera, may be coupled to device 600 through I/O functionality 611 . One or more aiming actuators 625 mechanically coupled to the camera may exchange signals with processor 601 through I/O function 611 .

As used herein, the term I/O refers generally to any program, operation or device that transfers data to or from system 600, as well as to or from peripheral devices. Each data transfer can be viewed as an output from one device and an input to another. Peripherals include input-only devices, such as keyboards and mice, output-only devices, such as printers, and devices that can both input and output, such as writable CD-ROMs. The term "peripheral device" includes both external devices, such as a mouse, keyboard, printer, monitor, microphone, game controller, video camera, external compact drive, or scanner, and internal devices, such as a CD-ROM drive, CD-R drive Or internal modem or other peripherals like flash reader/writer, hard drive.

In some embodiments of the invention, device 600 may be a video game unit, which may include controller 630 coupled to the processor via I/O function 611 either by wire (such as a USB cable) or wirelessly. Controller 630 may have an analog joystick control 631 and conventional buttons 633, providing control signals commonly used during video game play. Such video games may be implemented as processor-readable data and/or instructions from program 604 , which may be stored in memory 602 or other processor-readable media, such as one associated with mass storage 615 .

Joystick control 631 may generally be configured such that moving the joystick left or right produces a signal for movement along the X-axis, and moving it forward (up) or backward (down) produces a signal for movement along the Y-axis. In a joystick configured for three-dimensional motion, twisting the joystick left (counterclockwise) or right (clockwise) produces a signal that moves along the Z axis. These three axes - X, Y and Z - are commonly referred to as roll, pitch and yaw respectively, especially with regard to aircraft.

In addition to conventional features, controller 630 may include one or more inertial sensors 632 that may provide position and/or orientation information to processor 601 via inertial signals. The orientation information may include angular information such as pitch, roll or yaw of the controller 630 . By way of example, inertial sensors 632 may include any number and/or combination of accelerometers, gyroscopes, or tilt sensors. In a preferred embodiment, inertial sensors 632 include a tilt sensor adapted to sense the orientation of the joystick controller about the pitch and roll axes; a first accelerometer adapted to sense acceleration along the yaw axis; and a second accelerometer , suitable for sensing angular acceleration about the yaw axis. An accelerometer may be implemented, for example, as a MEMS device comprising a mass mounted by one or more springs, with sensors for sensing displacement of the mass relative to one or more directions. The mass displacement dependent signal from the sensor can be used to determine the acceleration of the joystick controller 630 . This technique can be implemented by instructions in a game program 604 that can be stored on the memory 602 and executed by the processor 601 .

By way of example, an accelerometer suitable as inertial sensor 632 could be a simple mass elastically coupled to the frame at three or four points, such as by springs. The pitch and roll axes lie in a plane that intersects the frame that is mounted to the joystick controller 630 . As the frame (and joystick controller 630) rotates about the pitch and roll axes, the mass will displace under the influence of gravity and the spring will elongate or compress in a manner dependent on the pitch and/or roll angle. The displacement of the mass can be sensed and converted into a signal dependent on the amount of pitch and/or roll. Angular acceleration about the yaw axis or linear acceleration along the yaw axis can also produce characteristic modes of spring compression and/or elongation or movement of the mass which can be sensed and converted into angular or linear acceleration dependent amount of signal. Such an accelerometer arrangement can measure pitch, roll angular acceleration about the yaw axis, and linear acceleration along the yaw axis by tracking mass movement or spring compression and expansion forces. There are many different ways to track the position of the block and/or the force exerted on it, including resistive strain gauge materials, optical sensors, magnetic sensors, Hall effect devices, piezoelectric devices, capacitive sensors, and the like.

Additionally, joystick controller 630 may include one or more light sources 634, such as light emitting diodes (LEDs). Light source 634 may be used to distinguish one controller from the other. For example, one or more LEDs can be coded to achieve this by blinking or holding on to an LED pattern. By way of example, 5 LEDs may be provided on the joystick controller 630 in a linear or two-dimensional pattern. Although a linear arrangement of LEDs is preferred, the LEDs may alternatively be arranged in a rectangular pattern or an arcuate pattern to facilitate determining the image plane of the LED array when analyzing images of the LED pattern obtained by the image capture unit 623 . Additionally, the LED pattern code may also be used to determine the positioning of the joystick controller 630 during game play. For example, LEDs can help identify the controller's pitch, yaw, and roll. This detection mode can help provide a better user experience in games, such as aircraft flying games and so on. Image capture unit 623 may capture an image including joystick controller 630 and light source 634 . Analysis of these images can determine the position and/or orientation of the joystick controller. These analyzes may be implemented by program code instructions 604 stored in memory 602 and executed by processor 601 . In order to facilitate the image capturing unit 623 to capture the image of the light source 634, the light source 634 may be placed on two or more different sides of the joystick controller 630, such as the front and the rear (shown as dotted lines). This placement allows image capture unit 623 to obtain images of light source 634 for different orientations of joystick controller 630 depending on how the user holds joystick controller 630 .

Additionally, the light source 634 may provide a telemetry signal to the processor 601, eg, in the form of pulse code, amplitude modulation or frequency modulation. Such telemetry signals may indicate which buttons of the joystick are being pressed and/or to what extent the buttons are pressed. The telemetry signal can be encoded into the optical signal, for example, by pulse encoding, pulse width modulation, frequency modulation or light intensity (amplitude) modulation. Processor 601 may decode the telemetry signal from the light signal and execute game commands in response to the decoded telemetry signal. The telemetry signal can be decoded by analyzing the image of the joystick controller 630 obtained by the image capture unit 623 . Optionally, device 600 may include a separate light sensor dedicated to receiving telemetry signals from light source 634 . The use of LEDs in connection with determining the amount of brightness in interfacing with a computer program is described, for example, in Richard L. Marks et al., entitled "Computer Image and Audio processing of Intensity and Input Devices for Interfacing With A Computer Program" (Attorney Docket No. SONYP052 ), described in US Patent Application No. 11/429,414, filed May 4, 2006, the entire contents of which are incorporated herein by reference. Additionally, analyzing images containing light source 634 may be used both for telemetry and to determine the position and/or orientation of joystick controller 630 . This technique may be implemented by instructions of a program 604 , which may be stored in memory 602 and executed by processor 601 .

The processor 601 may use the inertial signal from the inertial sensor 632 in combination with the light signal from the light source 634 detected by the image capture unit 623 and/or the sound source position and characteristic information from the sound signal detected by the microphone array 622 to infer Information about the location and/or orientation of the controller 630 and/or its user. For example, "acoustic radar" sound source locations and characteristics may be used in conjunction with microphone array 622 to track moving sounds while joystick controller movement is independently tracked (by inertial sensors 632 and or light sources 634). In acoustic radar selection, a pre-calibrated listening zone is selected at runtime, and sounds from sources outside the pre-calibrated listening zone are filtered out. The pre-calibrated listening zone may include a listening zone corresponding to the focal volume or field of view of the image capture unit 623 . Examples of acoustic radar are described in detail in U.S. Patent Application No. 11/381,724, entitled "METHOD AND APPARATUS FOR TARGETED SOUND DETECTION AND CHARACTERIZATION," by Xiadong Mao, filed May 4, 2006, which is incorporated by reference into here. Any number of different combinations of different modes of providing control signals to processor 601 may be used in conjunction with embodiments of the present invention. This technique may be implemented by program code instructions 604 storable in memory 602 and executed by processor 601, and may optionally include controlling one or more processors to select a pre-calibrated listening zone at runtime and filter One or more instructions to drop sound from sources outside the pre-calibrated listening zone. The pre-calibrated listening zone may include a listening zone corresponding to the focal volume or field of view of the image capture unit 623 .

Program 604 may optionally include one or more instructions to instruct one or more processors to generate a discrete-time domain input signal x _m (t) from microphones M _o ... M _m of microphone array 622 to determine the listening fan region and use this listening sector in semi-blind source separation to select finite impulse response filter coefficients for separating different sound sources from the input signal x _m (t). Program 604 may also include instructions to apply one or more fractional delays to selected input signals x m (t) in addition to the input signal x _o (t) from the reference microphone _{M o} . Each fractional delay can be chosen to optimize the signal-to-noise ratio of the discrete-time domain output signal y(t) from the microphone array. The fractional delay may be chosen such that the signal from the reference microphone M _o is first in time relative to the signals from the other microphone(s) in the array. Program 604 may also include instructions to introduce a fractional time delay Δ into the output signal y(t) of the microphone array such that: y(t+Δ)=x(t+Δ)*b _o +x(t−1+ Δ)*b ₁ +x(t-2+Δ)*b ₂ +...+x(t-N+Δ)b _N , where Δ is between 0 and ±1. An example of this technique is described in detail in Xiadong Mao, US Patent Application No. 11/381,729, entitled "ULTRA SMALL MICROPHONE EARRAY," filed May 4, 2006, the entire disclosure of which is incorporated herein by reference.

Program 604 may include one or more instructions that, when executed, cause system 600 to select a pre-calibrated listening sector containing a sound source. These instructions may cause the device to determine whether the sound source is located in the initial sector or on a particular side of the initial sector. These commands may, when executed, select a different sector on a particular side of the default sector if the sound source is not in the default sector. The different sectors can be characterized by the attenuation of the input signal closest to the optimum. These instructions may, when executed, calculate the attenuation of the input signal from the microphone array 622 and bring the attenuation to an optimal value. These instructions may, when executed, cause device 600 to determine attenuation values for the incoming signal for one or more sectors, and to select the sector with the attenuation closest to the optimal value. Examples of such techniques are described, for example, in U.S. Patent Application No. 11/381,725, entitled "METHODS ANDAPPARATUS FOR TARGETED SOUND DETECTION," by Xiadong Mao, filed May 4, 2006, the disclosure of which is incorporated herein by reference .

Signals from inertial sensors 632 may provide one portion of the tracking information input, and signals generated by image capture unit 623 from tracking one or more light sources 634 may provide another portion of the tracking information input. By way of example, and not limitation, this "mixed mode" signal could be used in a football type video game where a quarterback throws the ball to the right after pretending to move his head to the left. In particular, a game player holding the controller 630 can turn his head to the left and make a sound when swinging the controller to the right as if it were a football to make a throwing motion. Microphone array 622 combined with "acoustic radar" program code can track the user's voice. The image capture unit 623 may track the movement of the user's head or track other commands that do not require voice or use of the controller. Sensor 632 may track the movement of a joystick controller (representing a football). Image capture unit 623 may also track light source 634 on controller 630 . The user may command the release of the "ball" when a certain amount and/or direction of acceleration of the joystick controller 630 is reached, or by pressing a button on the controller 630 to trigger a key.

In some embodiments of the invention, inertial signals, such as from accelerometers or gyroscopes, may be used to determine the position of controller 630 . In particular, the acceleration signal from the accelerometer can be integrated once over time to determine a change in velocity, and the velocity can be integrated over time to determine a change in position. If the values of initial position and velocity at a certain moment are known, these values and changes in velocity and position can be used to determine absolute position. Although position determination using inertial sensors can be made faster than using image capture unit 623 and light source 634, inertial sensor 632 can be subject to a type of error known as "drift," where errors accumulated over time can lead to The difference D between the calculated position of the joystick 631 (shown in dashed lines) and the actual position of the joystick controller 630 . Embodiments of the present invention allow for a variety of methods to handle this error.

Drift can be manually removed, for example, by resetting the initial position of the controller 630 equal to the current calculated position. The user may utilize one or more buttons on the controller 630 to trigger commands to reset the initial position. Alternatively, image-based drifting can be achieved by resetting the current position to a position determined from an image obtained from the image capture unit 623 as a reference. Such image-based drift compensation may be accomplished manually, such as when a user activates one or more buttons on joystick controller 630 . Optionally, image-based drift compensation can be implemented automatically, eg, at regular intervals or in response to game play. This technique may be implemented by program code instructions 604 , which may be stored in memory 602 and executed by processor 601 .

In some embodiments it may be desirable to compensate for spurious data in the inertial sensor signal. For example, the signal from the inertial sensor 632 may be oversampled, and a running average may be calculated from the oversampled signal to eliminate artifacts in the inertial sensor signal. In some cases it may be desirable to oversample the signal and discard high and/or low values from some subset of the data points and compute a moving average over the remaining data points. Additionally, other data sampling and processing techniques can be used to condition the signals from the inertial sensors to eliminate or reduce the effects of spurious data. The choice of technique may depend on the characteristics of the signal, the calculations to be performed on the signal, the characteristics of the game being played, or a combination of two or more of these factors. This technique may be implemented by instructions of a program 604 , which may be stored in memory 602 and executed by processor 601 .

Processor 601 may perform analysis of inertial signal data 606 in response to inertial signal data 606 and program code instructions of program 604 stored and retrieved by memory 602 and executed by processor module 601 as described above. The code portion of program 604 may conform to any of a number of different programming languages, such as assembly, C++, JAVA or other languages. The general-purpose computer formed by the processor module 601 becomes a special-purpose computer when a program such as the program code 604 is executed. Although the program code 604 described herein is implemented in software and runs on a general-purpose computer, those skilled in the art will appreciate that the method of task management can alternatively utilize hardware, such as an application-specific integrated circuit (ASIC) Or other hardware circuits to achieve. In this way, it should be understood that all or part of the embodiments of the present invention may be realized by software, hardware or some combination thereof.

In one of these embodiments, the program code 604 may include a set of processor-readable instructions that implement a method having the same characteristics as the method 510 in FIG. 5B and the method 520 in FIG. 5C or both or both of them. A certain combination of a plurality of the same characteristics. Program code 604 may generally include one or more instructions that instruct one or more processors to analyze signals from inertial sensors 632 to generate position and/or orientation information and utilize such information during video game play.

Program code 604 may optionally include processor-executable instructions comprising one or more instructions that, when executed, cause image capture unit 623 to monitor a field of view in front of image capture unit 623, identifying one or more light sources within the field of view 634. Detect a change in light emitted by the light source(s) 634; and trigger an input command to the processor 601 in response to the detected change. The use of LEDs in conjunction with image capture devices to trigger the motion of a game controller is described, for example, in U.S. Patent Application No. 10/759,782, filed January 16, 2004, by Richard L. Marks, entitled "METHOD AND APPARATUS FOR LIGHT INPUT DEVICE" described in , which is hereby incorporated by reference in its entirety.

Program code 604 may optionally include processor-executable instructions comprising one or more instructions that, when executed, use signals from inertial sensors and signals generated by image capture units from tracking one or more light sources as input to the gaming system , for example, as described above. Program code 604 may optionally include processor-executable instructions including one or more instructions that, when executed, compensate for drift in inertial sensor 632 .

Although embodiments of the present invention have been described by way of example in relation to gaming with video game controller 630, embodiments of the present invention, including system 600, may be used with inertial sensing capabilities and inertial sensor signal transmission capabilities, wireless or otherwise. , any user-manipulated body, molded object, knob, structure, etc.

By way of example, embodiments of the invention may be implemented on a parallel processing system. Such parallel processing systems typically include two or more processor elements configured to execute portions of a program in parallel using separate processors. By way of example, and not limitation, Figure 7 shows one type of cell processor 700 according to one embodiment of the present invention. This cell processor 700 can be used as the processor 601 in FIG. 6 or the processor 502 in FIG. 5A. In the example depicted in FIG. 7 , a cell processor 700 includes a main memory 702 , a power processor element (PPE) 704 , and a plurality of co-processor elements (SPE) 706 . In the example depicted in FIG. 7, cell processor 700 includes a single PPE 704 and eight SPEs 706. In this configuration, seven SPEs 706 can be used for parallel processing, and one can be kept as a backup in case one of the other seven fails. A cell processor may optionally include multiple sets of PPEs (PPE sets) and multiple sets of SPEs (SPE sets). In this case, hardware resources can be shared among the units in a group. However, the SPE and PPE must appear to the software as independent components. As such, embodiments of the present invention are not limited to employing the configuration shown in FIG. 7 .

Main memory 702 typically includes general purpose nonvolatile memory, and special purpose hardware registers or arrays for functions such as system configuration, data transfer synchronization, memory mapped I/O, and I/O subsystems. In an embodiment of the invention, video game program 703 may reside in main memory 702 . Memory 702 may also contain signal data 709 . Video program 703 may include inertial, image and acoustic analyzers and mixers configured as described above with reference to Figures 4, 5A, 5B or 5C, or some combination thereof. Program 703 may run on the PPE. Program 703 may be divided into multiple signal processing tasks so as to be run on the SPE and/or PPE.

By way of example, PPE 704 may be a 64-bit PowerPC processor unit (PPU) with associated caches L1 and L2. PPE 704 is a general purpose processing unit that can access system management resources (such as memory protection tables, for example). Hardware resources can be explicitly mapped into real address space as seen by the PPE. Thus, the PPE can directly address any of these resources with the appropriate effective address value. A primary function of the PPE 704 is to manage and distribute tasks for the SPE 706 in the cell processor 700.

Although only one PPE is shown in FIG. 7, certain cell processor implementations, such as the Cell Broadband Engine Architecture (CBEA), cell processor 700 may have multiple PPE groups organized into multiple PPE groups where there may be more than one PPE. PPE. These PPE groups may share access to main memory 702 . In addition, cell processor 700 may include two or more SPE groups. These groups of SPEs may also share access to main memory 702 . These configurations are within the scope of the present invention.

Each SPE 706 includes a co-processor unit (SPU) and its own local storage area LS. The local storage LS may comprise one or more separate areas of memory storage, each associated with a particular SPU. Each SPU can be configured to execute instructions (including data load and data store operations) only from its own associated local storage area. In this configuration, data transfers between the local storage LS and the rest of the system 700 can be performed by issuing direct memory access (DMA) commands from a memory flow controller (MFC) to or from (individual SPE ) local storage area to transfer data. SPUs are less complex computing units than PPE 704 in that they do not perform any system management functions. SPUs are generally single instruction, multiple data (SIMD) capable and typically process data and initiate any required data transfers (subject to access attributes established by the PPE) to perform their assigned tasks. The purpose of the SPU is to enable applications that require higher computing unit density and can efficiently utilize the provided instruction set. The large number of SPEs in the system managed by PPE 704 allows for cost-effective processing in a large range of applications.

Each SPE 706 may include a dedicated memory flow controller (MFC) including an associated memory management unit capable of storing and processing memory protection and access permission information. MFC provides the main methods for data transfer, protection, and synchronization between the cell processor's main memory and the SPE's local storage. MFC commands describe the transfer to be performed. Commands used to transfer data are sometimes called MFC Direct Memory Access (DMA) commands (or MFC DMA commands).

Each MFC can support multiple DMA transfers simultaneously and can maintain and process multiple MFC commands. Each MFC DMA data transfer command request can include both a Local Storage Address (LSA) and an Effective Address (EA). A local storage address can directly address only the local storage area of the SPE it is associated with. Effective addresses can have more general use, eg, it can refer to main memory, including all SPE local storage areas, if they are mapped into the actual address space.

To facilitate communication between SPEs 706 and/or between SPEs 706 and PPE 704, SPEs 706 and PPE 704 may include signaling registers related to signaling events. PPE 704 and SPE 706 can be coupled through a star topology, wherein PPE 704 sends messages to SPE 706 as a router. Optionally, each SPE 706 and PPE 704 may have a one-way signaling register called a mailbox. Mailboxes can be used by SPE 706 for host operating system (OS) synchronization.

Cell processor 700 may include input/output (I/O) functionality 708 by which cell processor 700 may communicate with peripheral devices, such as microphone array 712 and optional image capture unit 713 and game controller 730 to interact. The game controller unit may include an inertial sensor 732 , and a light source 734 . Additionally component interconnect bus 710 may connect the various components listed above. Each SPE and PPE can access the bus 710 through the bus interface unit BIU. Cell processor 700 may also include two controllers typically found in processors: the memory interface controller MIC, which controls data flow between bus 710 and main memory 702, and the bus interface controller BIC, which controls the I Data flow between /O 708 and bus 710. Although the requirements for the MIC, BIC, BIU and bus 710 may vary considerably for different applications, those skilled in the art will be familiar with their functions and circuits to implement them.

The cell processor 700 may also include an internal interrupt controller IIC. The IIC component manages the priority of interrupts submitted to the PPE. The IIC allows interrupts from other components of the cell processor 700 to be handled without using the main system interrupt controller. The IIC can be viewed as a second-level controller. The main system interrupt controller can handle interrupts initiated from the outside to the cell processor.

In embodiments of the invention, certain computations, such as the fractional delay described above, may be performed in parallel using PPE 704 and/or one or more SPEs 706. Each fractional delay computation may run as one or more separate tasks that different SPEs 706 may execute as they become available.

While the above is a complete description of the preferred embodiments of the invention, various alternatives, modifications and equivalents may be used. The scope of the invention, therefore, should be determined not with reference to the foregoing description, but should instead be determined with reference to the appended claims, along with their full scope of equivalents. Any feature described herein, whether preferred or not, may be combined with any other feature described herein, whether preferred or not. In the following claims, the indefinite article "a" or "an" refers to a quantity of one or more of the item following the article, unless expressly stated otherwise. The appended claims should not be interpreted as including means-plus-function limitations unless expressly defined in a given claim using the phrase "means for".

Claims (15)

1. one kind is used to analyze the system that game control is imported data, comprising:

Image dissector is configured to receive and analyze the input data from image capture apparatus, the input control data;

The inertia analyzer is configured to receive and analyze the input data from inertial sensor;

The acoustic analysis device is configured to receive and analyze the input data from acoustic receivers; And

Mixer, be configured to receive one or more input data by analysis, and produce trace information based on the information of the one or more receptions from inertia analyzer, image dissector harmony credit parser from inertia analyzer, the image dissector harmony credit parser.

2. the system as claimed in claim 1 also comprises the gesture recognition device, this gesture recognition device be configured to based on above-mentioned trace information be associated in the game environment at least one the action with one or more user actions.

3. the system as claimed in claim 1, also comprise game console, wherein game console comprises that at least one operationally is couple to the inertial sensor of above-mentioned inertia analyzer, and/or one or more light sources that are configured to provide one or more light signals, and/or one or more acoustic signal generator.

4. system as claimed in claim 3, wherein above-mentioned at least one inertial sensor comprises at least one in accelerometer or the gyroscope.

5. the system as claimed in claim 1 also comprises the image capture apparatus that is couple to image dissector.

6. system as claimed in claim 5, wherein said image capture apparatus comprises video camera.

7. the system as claimed in claim 1 also comprises the microphone array that operationally is couple to acoustic analysis device unit.

8. the system as claimed in claim 1, wherein mixer also is configured to one or more by analysis the input data allocations one or more distribution values that receive of mixer from inertia analyzer, image dissector harmony credit parser.

9. the system as claimed in claim 1, wherein said trace information comprises that the user can handle the position and/or the directional information of object.

10. system as claimed in claim 9, wherein position and/or directional information comprise the indication user can handle facing upward of object bow, go off course and roll at least one information.

11. system as claimed in claim 9, wherein position and/or directional information comprise the information of indication about the position of one or more coordinates.

12. system as claimed in claim 9, wherein position and/or directional information comprise the information of indication about the speed of one or more coordinates.

13. system as claimed in claim 9, wherein position and/or directional information comprise the information of indication about the acceleration of one or more coordinates.

14. system as claimed in claim 9, wherein the user can to handle object be the controller that is used for video game system.

15. a method of analyzing game control input data comprises:

Receive the input control data from one or more input medias, described input control data is associated with one or more user actions;

Be the input control data assignment profile value that receives;

Based on the distribution value of distributing to the input control data that receives, identification is associated with one or more user actions, at least one action in game environment;

And

In game environment, carry out above-mentioned at least one action.

Images