KR101060779B1 - Methods and apparatuses for applying gearing effects to an input based on one or more of visual, acoustic, inertial, and mixed data - Google Patents

Abstract

In accordance with the present invention, methods and apparatuses for interacting with a computer game system are provided. The computer game system includes a video capture device for capturing image data. The method includes displaying an input device with a video capture device, the input device comprising: positioning of the input device to be interpreted by the computer game system based on the analysis of the captured image data and the state of the plurality of lights; It has a plurality of lights that are modulated to convey communication data. The method also includes defining an object movement of the computer game to be executed by the computer game system such that the movement of the object is mapped to movements in the position of the input device as detected in the captured image data. The method then establishes gearing between the object movement of the computer game and the positional movement of the input device. Gearing defines the ratio between positional movements of the input device and movements of the object. Gearing may be set dynamically by the game and the user, or may be preset by the user or software configured according to the gearing algorithm. Embodiments of the input device may be used to provide gearable inertial data and may be interpreted by a computer game system. In addition, embodiments may use acoustic detection systems, as well as a mix of image capture, inertial capture, and acoustic capture, to apply the desired gearing.

Computer Game System, Image Data, Gearing, Mapping

Description

Methods and apparatus for applying gearing effects to input based on one or more of visual, acoustic, inertial, and mixed data}

The video game industry has seen many changes over the years. As computing power expands, video game developers have similarly produced game software that takes advantage of these increases in computing power. Because of this, video game developers have coded games that incorporate delicate operations and maths to get a very realistic game experience.

Exemplary gaming platforms include Sony Playstation or Sony Playstation2 (PS2) sold in the form of a game console. As is known, game consoles are designed to be connected to a monitor (typically a television) and to enable user interaction via handheld controllers. The game console may include a CPU, a graphics synthesizer for processing intensive graphics operations, professional processing hardware including a vector unit for performing geometry transformations, and other removable hardware. , Firmware, and software. The game console is also designed with an optical disc tray for receiving game compact discs for local play through the game console. Online games are also possible, where users can interact and play with other users over the Internet.

As game complexity continues to inspire players, game software and hardware manufacturers have continued to innovate to enable additional interactions. In fact, however, the way users interact with games has not changed dramatically over the years. Typically, users still play computer games using handheld controllers or interact with programs using mouse pointing devices.

As mentioned above, there is a need for methods and systems that enable a user to have more advanced interaction with game play.

Collectively, the present invention satisfies this need by providing methods, systems, and apparatuses that enable dynamic user interaction with a computing system. In one embodiment, the computing system will execute the program and the interaction will be with the program. The program may for example be a video game defining interactive objects or features. The interaction includes providing users with the ability to adjust gearing elements to adjust the extent to which the processing is to be performed. In one embodiment, gearing may be applied to the relative movement between the movement of the input device and the amount of movement processed by the object or feature of the computer program.

In yet another embodiment, gearing may be applied to the features of the computer program and the detection from the input device may be based on the processing of the inertial analyzer. The inertial analyzer will track the input device for inertial motion, which can then pass this information to the program. The program then obtains the output from the inertial analyzer so that the gearing amount can be applied to the output. The amount of gearing will then indicate the rate at which the program will compute the action. The operation may take any number of forms, and one embodiment of the operation may be to generate noise, various odors, movement by an object, or a variable. If the output is a variable, a variable (eg, multiplier or the like) can be used to complete the execution of the process, so that the process will take into account the amount of gearing. The gearing amount can be preset by the user, set dynamically or can be adjusted as required.

In one embodiment, tracking can be performed by an acoustic analyzer. The acoustic analyzer is configured to receive acoustic signals from the input device, and the acoustic analyzer can deliver the amount of gearing to be applied with the command or interaction to be performed. The acoustic analyzer may be in the form of a computer program segment (s) or may be specifically defined in circuitry designed to process acoustic signal information. Therefore, the acoustic signal information can be set dynamically by a program and gearing data that can be set according to a request by a user through an input device (for example, by selecting a button of a controller or by a voice command or the like). It may include.

In one embodiment, tracking of the input device may be performed via an image analyzer. As discussed below, the image analyzer may include a camera that captures images of the space in which the user and the input device are located. In this embodiment, the image analyzer is determining the position of the controller to cause some respective action on the characteristics of the processing program. The program may be a game and the feature may be an object controlled by the input device. The image analyzer is further configured to mix the position data with the input gearing value. The gearing value may be provided by the user dynamically during execution or may be set by the program depending on the operation in the execution session. The gearing input will set the relative impact on some processing by the computer program based on the input action by the user. In one embodiment, gearing will translate the command or action from a user or user input device to a feature of the program. Features of the program need not be visible objects, but may include adjustment of variables used to calculate some parameters, evaluation of acoustic, vibration, or image movement or translation. Therefore, gearing will provide additional sensing of control in the interaction provided to the program and its features.

In yet another embodiment, a mixer analyzer is provided. The mixer analyzer is designed to create hybrid effects as a feature of the game. For example, the mixer analyzer can obtain input from a combination of an image analyzer, an acoustic analyzer, an inertial analyzer, and the like. Therefore, in one embodiment, the mixer analyzer may receive several gearing variables that may be mixed and synthesized to create interactions with hybrid results, instructions, or features of the program. Also, it is to be understood that the features of the program include visible objects, invisible objects, variables used in the processing of motion, adjustments to audible responses, and the like.

In one embodiment, the amount by which the movement by the user's controller will be translated into movement or motion relative to the features of the game will be at least partially related to the gearing set or determined. Gearing can be set dynamically during the game, preset or adjusted during game play by the user, and the response can be mapped to video game objects or features to provide another level of user interaction and enhanced experience. mapping).

The invention can be implemented in a variety of ways, including as a process, an apparatus, a system, a device, or a method. Several inventive embodiments of the present invention are described below.

In one embodiment, a method is provided for interacting with a computer game system. The computer game system includes a video game device for capturing image data. The method includes displaying an input device to a video capture device, the input device being based on the analysis of the captured image data and the state of the plurality of lights by means of positioning of the input device and a computer game system. It has a plurality of lights that are modulated to convey communication data to be interpreted. The method also includes defining a movement of the object of the computer game to be executed by the computer game system, wherein the movement of the object may be mapped to positional movements of the input device as detected in the captured image data. The method then establishes gearing between the movement of the object of the computer game and the positional movements of the input device. Gearing establishes the ratio between the positional movements of the input device and the movements of the object. Gearing may be set dynamically by the game or the user, or may be preset by the user or by software configured according to the gearing algorithm.

In yet another embodiment, a method for interacting with a computer game system is provided. The computer game system includes a video capture device for capturing image data. The method includes displaying an input device to a video capture device, the input device delivering communication data to be interpreted by a computer game system and positioning of the input device based on the analysis of the captured image data. . The method then defines an object movement of the computer game to be executed by the computer game system, which may be mapped to the position movement of the input device as detected in the captured image data. The method then establishes gearing between the object movement of the computer game and the positional movements of the input device. Gearing establishes a user adjustable ratio between the positional movements of the input device and the movements of the object.

In yet another embodiment, a computer readable medium is provided that includes program instructions for interacting with a computer game system. The computer game system includes a video capture device for capturing image data. The computer readable medium includes program instructions for detecting the display of the input device on the video capture device, the input device being interpreted by the computer game system and the positioning of the input device based on the analysis of the captured image data. Pass communication data. The computer readable medium also includes program instructions for defining object movements of the computer game to be executed by the computer game system, where the object movements are mapped to position movements of the input device as detected in the captured image data. The computer readable medium also includes program instructions for establishing gearing between the object movement of the computer game and the position movements of the input device, wherein the gearing is user adjustable between the position movements of the input device and the movements of the object. It can be configured to establish a ratio. The user adjustable rate may change dynamically during game play, before a game play session, or during certain motion events during the game.

In yet another embodiment, a system is provided for performing dynamic user interaction between user actions and actions to be performed by an object of a computer program. The system includes a computing system, a video capture device coupled to the computing system, and a display that receives input from the computing system. The system also includes an input device for interfacing with a computer program to be executed by the computing system. The input device has gearing control to establish a ratio between the movement of the controlled object and the movement of the mapped input device. Objects are defined by computer programs and executed by computing systems. The movement of the input device is identified by the video capture device, and the movement of the object displayed on the display has a gearing value set by the gearing control of the input device. In this embodiment, the gearing associated with the input device can be applied to a general application and need not necessarily be for a game.

In yet another embodiment, an apparatus is provided for performing dynamic user interaction between user actions and actions to be performed by an object of a computer program. The apparatus includes an input device for interfacing with a computer program to be executed by a computing system. The input device has gearing control to establish a ratio between the movement of the controlled object and the movement of the mapped input device. The object defined by the computer program and executed by the computing system, and the movement of the input device, is identified by the video capture device, and the movement of the object is displayed on the display with the gearing value set by the gearing control of the input device. do.

In yet another embodiment, gearing may be applied to the features of the computer program and the detection from the input device may be based on the processing of the inertial analyzer. The inertial analyzer will track the input device for inertial motion, which can then pass this information to the program. The program will then obtain the output from the inertial analyzer so that the gearing amount can be applied to the output. The amount of gearing will then indicate the rate at which the program calculates the motion. The operation may take any number of forms, and one embodiment of the operation may be to generate noise, various odors, movement by an object, or a variable. If the output is a variable, a variable (eg multiplier, etc.) can be used to complete the execution of the process, so that the process will consider the amount of gearing. The amount of gearing may be preset, dynamically set by the user, or adjusted as required.

In one embodiment, tracking can be performed by an acoustic analyzer. The acoustic analyzer is configured to receive acoustic signals from the input device, and the acoustic analyzer can deliver the amount of gearing to be applied to the command or interaction being performed. The acoustic analyzer may be in the form of computer program segment (s) or may be specifically defined in circuitry designed to process acoustic signal information. Therefore, the sound signal information can be set dynamically by a program and gearing data that can be set according to a request by a user through an input device (for example, by selecting a button on a controller or by a voice command or the like). It may include.

In yet another embodiment, a mixer analyzer is provided. The mixer analyzer is designed to create hybrid effects as a feature of the game. For example, a mixer analyzer can obtain input from a combination of an image analyzer, an acoustic analyzer, an inertial analyzer, and the like. Therefore, the mixer analyzer may, in one embodiment, receive several gearing variables that may be mixed and synthesized to create interactions with hybrid results, instructions, or features of the program. It is also to be understood that the features of the program include visible objects, invisible objects, variables used in the processing of motion, adjustments to audible responses, and the like.

Advantages of the present invention will become apparent from the following embodiments, for example in conjunction with the accompanying drawings which illustrate the principles of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention will be readily understood by the following embodiments in conjunction with the accompanying drawings, wherein like reference numerals refer to like elements.

1 illustrates an interactive game setup with an image capture device.

2 illustrates an example computer interaction using a video processing system.

3 is a block diagram of an example user input system for interacting with an object on a graphical display.

4 is a schematic block diagram of a computer processing system configured to implement embodiments of the present invention described herein.

5 is a block diagram of the configuration of components of a video game console suitable for use with an object being manipulated and functioning as an optional input device.

FIG. 6 is a block diagram illustrating functional blocks used to track and identify a group of pixels corresponding to a user input device, as manipulated by a user.

7 shows a schematic block diagram for an example image processing system.

8 illustrates an example application of the image processing system of FIG. 7.

9 and 10 are top and back views of example controllers interacting with an image capture device.

11 illustrates an example application for the controller of FIGS. 9 and 10.

FIG. 12 shows an exemplary software controlled gearing quantity for the application shown in FIG. 11.

13 shows an example controller with a steering wheel.

FIG. 14 illustrates an example application for the controller of FIG. 13.

FIG. 15 shows an example graph showing example changes in the amount of gearing over time.

16 illustrates another example application of an image processing system corresponding to user interaction.

FIG. 17 shows a graph showing an example gearing quantity at different times along the length of a baseball bat swing for the application shown in FIG. 16.

18 illustrates another example application of an image processing system corresponding to user interaction.

FIG. 19 shows a graph showing an exemplary change in the amount of gearing when “release” a football for the application shown in FIG. 18.

20 depicts a flowchart illustrating an example process for receiving user input to a computer program.

In the following examples, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without some or all of these specific details. Known procedures will not be described in detail so as not to obscure the present invention.

The techniques described herein can be used to provide dynamically geared inputs for interaction with a computer program. In general, gearing may be defined as an input that may have varying degrees in size and / or time. Then, the degree of gearing can be transferred to the computing system. The degree of gearing can be applied to the processes executed by the computing system. In analogy, a processor may be represented as a fluid in a bucket having inputs and outputs. The fluid is a process running in the system, so gearing controls the processing of an aspect performed by the computing system. As one example, gearing may control the rate at which the fluid is emptied from the fluid reservoir with respect to the input amount considered as droplets of fluid entering the bucket. Therefore, the filling speed is dynamic, the discharging speed is dynamic, and the discharging speed can be influenced by gearing. Therefore, the gearing can be adjusted or timed to match the change value that can be streamed into a program such as a game program. Gearing can also affect counters, such as movement data counters, which control motion or game elements, objects, players, characters, etc. by the processor.

If such an analogy is made in a more realistic computing example, the speed at which the fluid is emptied can be the speed at which control is passed or executed by the characteristics of the computer program, corresponding to the sum of some input and gearing. Features of the computer program may be objects, processes, variables, or scheduled / custom algorithms, characters, game players, mice (2D or 3D), and the like. The results of the processing, which can be changed by gearing, can be conveyed to the viewer in any number of ways. In one way, there may be visual on the display screen, audible by sound, vibration by touch, combinations thereof, etc., and may occur by altered response of processing to interactive elements of a game or program. have.

Input can be obtained by tracking performed through: (1) image analysis, (2) inertial analysis, (3) acoustic analysis, or hybrid mixed analysis of (1), (2) or (3). While various embodiments are provided in connection with image analysis and applied gearing, tracking is not limited to video, and may be accomplished by various methods, specifically, inertial analysis, acoustic analysis, mixtures thereof, and other suitable analyzers. It should be understood that it can.

In various embodiments, a computer or game system having a video camera (eg, image analysis) can process image data and identify various operations occurring in a given volume or area of focus in front of the video camera. Can be. Such operations typically include moving or rotating an object in three dimensional space, or operating various controls such as buttons, dials, joy sticks, and the like. In addition to these techniques, the technique of the present invention also scales, referred to herein as gearing, to adjust the sensitivity of the input in relation to a feature of a program or one or more corresponding operations on a display screen. ) Provides additional functionality for adjusting elements. For example, the actions on the display screen may be an object that is the focal point of the video game. An object may also be a feature of a program, such as a variable, multiplier, or operation to be rendered into sound, vibration, images on a display screen or combinations thereof, and other representations of geared output.

In another embodiment, gearing may be applied to the features of the computer program and the detection of the input device may be based on processing by an inertial analyzer. The inertial analyzer will track the input device for inertial motion, which can then pass information to the program. The program will then get the output from the inertial analyzer so that the amount of gearing can be applied to the output. The amount of gearing will then dictate the rate at which the program will compute the motion. The operation may take any number of forms, and one embodiment of the operation may be to generate noise, various odors, vibrations, movement by an object, or operation by a program that outputs visible and / or audible results. have. If the output is a variable, the variable can be used to complete the execution of the process, and the process will take into account the amount of gearing. The amount of gearing may be preset, dynamically set by the user, or adjusted as desired.

Various types of inertial sensor devices provide information about six degrees of freedom (e.g., X, Y, and Z translation (e.g., acceleration) and rotation about the X, Y, and Z axes). It can be used to. Embodiments of suitable inertial sensors to provide information about six degrees of freedom include accelerometers, one or more single axis accelerometers, mechanical gyroscopes, ring laser gyroscopes or these Combinations of two or more of them.

The signals from the sensor (s) can be analyzed to determine the motion and / or direction of the controller during play of the video game in accordance with the method of the present invention. This method may be embodied as a series of processor executable program code instructions stored in a processor readable medium and executed on a digital processor. For example, a video game system can include one or more processors. Each processor may be any suitable digital processor, for example a microprocessor of the type commonly used in video game consoles or custom multi-processor cores. In one embodiment, the processor may implement the inertial analyzer through execution of processor readable instructions. Portions of the instructions may be stored in memory. Optionally, the inertial analyzer can be implemented in hardware, for example, an application specific integrated circuit (ASIC) or a digital signal processor (DSP). Such analyzer hardware may be located in a controller or console, or may be located elsewhere in the distance. In hardware embodiments, the analyzer may, for example, be connected to signals from a processor or to a remote connection via a USB cable, Ethernet, network, Internet, short range wireless connection, broadband wireless, Bluetooth, or local network. It can be programmed in response to external signals from another source located.

The inertial analyzer may include or execute instructions that analyze the signals generated by the inertial sensors and use information regarding the position and / or orientation of the controller. Inertial sensor signals can be analyzed to determine information regarding the position and / or orientation of the controller. Location and / or orientation information may be used during the play of a video game with the system.

In one embodiment, the game controller may include one or more inertial sensors, which may provide position and / or direction information to the processor via an inertial signal. The direction information may include angle information such as tilt, roll or yaw of the controller. As mentioned above, for example, inertial sensors may include any number and / or combination of accelerometers, gyroscopes, or tilt sensors. In one embodiment, the inertial sensors are tilt sensors suitable for sensing the direction of the joystick controller with respect to the tilt and roll axes, a first accelerometer suitable for sensing acceleration along the yaw axis, and angular acceleration with respect to the yaw axis. A second accelerometer suitable for sensing. The accelerometer may be implemented as a MEMS device, for example comprising a mass mounted by one or more springs with sensors for sensing a displacement of the mass in relation to one or more directions. Signals from sensors that depend on the displacement of the mass can be used to determine the acceleration of the joystick controller. Such techniques may be implemented by instructions from a game program or a general program, which may be stored in memory and executed by a processor.

For example, an accelerometer suitable as an inertial sensor can be, for example, a simple mass elastically coupled at three or four points in the frame by springs. The pitch and roll axes lie in a plane that intersects the frame mounted to the joystick controller. As the frame (and joystick controller) rotates about the pitch and roll axes, the mass will move under the influence of gravity and the springs will extend or compress in a manner that depends on the angle of the pitch and / or roll. The displacement of the mass can be sensed and converted into a signal that depends on the pitch and / or amount of roll. Angular acceleration about the yaw axis or linear acceleration along the yaw axis produces characteristic patterns of compression and / or extension of springs or mass movement that can be detected and converted into signals depending on the amount of angular or linear acceleration. can do. Such accelerometer devices can measure tilt around the yaw axis, roll angular acceleration and linear acceleration along the yaw axis by tracking the movement of mass or compression and expansion forces of the springs. Numerous other ways of tracking the force on the location and / or mass of the mass, including strain-resistant metering materials, light sensors, magnetic sensors, hall-effect devices, piezoelectric devices, capacitive sensors, and the like have.

In addition, the light sources can provide telemetry signals to the processor, for example, in a pulse code, amplitude modulation or frequency modulation format. These telemetry signals can indicate which buttons are being pressed and / or how hard these buttons are being pressed. Telemetry signals may be encoded into an optical signal by, for example, pulse coding, pulse width modulation, frequency modulation, or light intensity (amplitude) modulation. The processor may decode the telemetry signal from the optical signal and execute game instructions in response to the decoded telemetry signal. The telemetry signals can be decoded from the analysis of the images of the joystick controller obtained by the image capture unit. Optionally, the apparatus may include a separate light sensor dedicated to receiving telemetry signals from light sources.

The processor uses the inertial signals from the inertial sensor together with the sound source position and characteristic information from the optical signals from the light sources detected by the image capturing unit and / or the acoustic signals detected by the microphone array. And / or infer information about the user's location and / or orientation. For example, the "sound radar" sound source position and characteristics may be used with the microphone array to track the moving voice while the movement of the joystick controller is tracked independently (via inertial sensors and / or light sources). Can be. In the acoustic radar, a predetermined listening area is selected at run time, and sounds generated from external sources of the predetermined listening area are filtered out. The predetermined listening areas may include a listening area corresponding to a focal volume or field of the image capturing unit.

In one embodiment, tracking can be performed by an acoustic analyzer. The acoustic analyzer is configured to receive acoustic signals from the input device, and the acoustic analyzer can deliver the amount of gearing to be applied to the command or interaction being performed. The acoustic analyzer may be in the form of computer program segment (s) or may be specifically defined in circuitry designed to process acoustic signal information. Therefore, the acoustic signal information is dynamically set by the program and receives gearing data that can be set according to a request by a user through an input device (for example, by selecting a button on a controller or by a voice command or the like). It may include. One embodiment of an acoustic analyzer is described in US patent application “Selective sound source with computer interaction processing” filed May 4, 2006 (Xiadong Mao, Richard L. Marks and Gary). Gary M. Zalewski; agent reference SCEA04005JUMBOUS.

Analyzers can be configured with a mapping chain. The mapping chains can be exchanged by the game during game-play as can be set with the analyzer and the mixer.

In one embodiment, tracking of the input device may be performed by an image analyzer. The image analyzer may include a camera that captures images of the space in which the user and the input device are located, as discussed below. In one embodiment, the image analyzer is determining the location of the controller that causes each action as a feature of the processing program. The program may be a game and the feature may be an object controlled by the input device. The image analyzer is also configured to mix the position data with the input gearing values. The gearing value may be provided by the user dynamically during execution, or may be set by a program that depends on the actions in the execution session. The gearing input will set the relative impact on some processing by the computer program based on the input action by the user. In one embodiment, gearing will translate the commands or actions from the user or user input device into features of the program. Features of the program need not be visible objects, but may include adjustment of variables used to calculate some parameters, evaluation or translation of acoustic, vibration, or image movement. Therefore, gearing will provide additional sensing of control over the interaction provided by the program and its features.

In yet another embodiment, a mixer analyzer is provided. The mixer analyzer is designed to create hybrid effects as a feature of the game. For example, the mixer analyzer can obtain input from a combination of an image analyzer, an acoustic analyzer, an inertial analyzer, and the like. Therefore, the mixer analyzer can receive several gearing variables that can be mixed and synthesized to produce an interaction with the hybrid result, instruction or program feature. In addition, the features of the program should be understood to include visible and invisible objects, variables used in the processing of motion, adjustments to audible responses, and the like.

1 illustrates an interactive game setup 100, in accordance with an embodiment of the present invention. The interactive game setup 100 includes a computer 102 called a “console” that is connected to the display screen 110. The image capture device 105 may be located above the display screen 110 and connected to the computer 102. In one embodiment, computer 102 is a game system console that enables users to play video games and interface with video games through controllers 108. Computer 102 may also be connected to the Internet to enable interactive online gaming. Although image capture device 105 is shown positioned above display screen 110, image capture device 105 may be any other that enables capturing images located approximately in front of display screen 110. It may be located in an approximate location. Techniques for capturing such movements and interactions are various, and exemplary techniques are described in British patent applications GB 0304024.3 (PCT / GB2004 / 000693) and GB 0304022.7 (PCT / GB2004 / 000703, filed February 21, 2003). ).

In one embodiment, the image capture device 105 may be as simple as a standard web-cam or may include more advanced technology. The image capture device 105 may capture images, digitize the images, and transfer the image data to the computer 102. In some embodiments, the image capture device will have integrated logic to perform digitization, and another embodiment where the image capture device 105 will deliver an analog video signal to the computer 102 for digitization. In either case, the image capture device 105 may capture color or black and white images of an object located in front of the image capture device 105.

2 illustrates an example computer interaction using an image capture and processing system. Computer 102 receives image data from image capture device 105 that generates images from field of view 202. The field of view 202 includes a user 210 for manipulating an object 215 in the shape of a toy plane. Object 215 includes a plurality of LEDs 212 that can be seen by image capture device 105. LEDs 212 provide location and orientation information of object 215. In addition, object 215 may include one or more actuation elements, such as buttons, triggers, dials, and the like, for generating computer input. Voice commands can also be used. In one embodiment, object 215 includes circuitry 220 including logic to modulate LEDs 212 for transferring data to computer 102 through image capture device 105. The data may comprise encoded instructions generated in response to operations of the actuating means. As the object 215 is moved and rotated in a three-dimensional space that can be seen by the image capture device 105, the LED positions represented in the image data are three-dimensional, as described with reference to FIGS. 3 to 6. Coordinates in the space, as well as direction information using α, β, and γ values, are translated into coordinates describing position information using x, y and z coordinates. The coordinates are passed to a computer application where the toy aircraft may be an aircraft game represented on the display 110 as a real or realistically moving aircraft 215 'that performs various stunts in response to manipulation of the toy aircraft 215. . Optionally, instead of the toy aircraft 215, the user 210 can operate the controller 108 (shown in FIG. 1), the movement of the controller 108 can be tracked, and the movement of the object on the display screen. Instructions can be captured to cause movement.

In one embodiment, the user 210 may choose to change or change the degree of interaction with the aircraft 215 'that is moving in the real world. The degree of interaction is such that the user 215 adjusts the amount of "gearing" that the user's controller 108 (or toy aircraft 215) will adjust to the amount that will be mapped to the movement by the aircraft 215 'which is in motion. By making it possible. Reactions that are mapped to realistically moving aircraft 215 '(e.g., video game objects), depending on the gearing that can be set dynamically, preset for the game, or adjusted by the user 210 during gameplay. Will change to provide another level of user interaction and an improved experience. Further details regarding gearing will be provided below with reference to FIGS. 7-20.

3 is a block diagram of an example user input system for interacting with an object on a graphical display that can be used to practice embodiments of the present invention. As shown in FIG. 3, the user input system consists of a video capture device 300, an input image processor 302, an output image processor 304, and a video display device 306. Video capture device 300 may be any device capable of capturing sequences of video images, and in one embodiment is a digital video camera (such as a “web-cam”), or a similar image capture device.

The video capture device 300 may be configured to provide a depth image. In the detailed description, the terms “depth camera” and “three-dimensional camera” refer to any camera capable of obtaining distance or depth information as well as two-dimensional pixel information. For example, the depth camera may use controlled infrared light to obtain distance information. Another example depth camera may be a stereo camera pair that triangulates distance information using two standard cameras. Similarly, the term “depth sensing device” refers to any type of device capable of obtaining distance information as well as two-dimensional pixel information.

Thus, camera 300 may provide the ability to capture and map a third dimension in addition to a standard two-dimensional video image. Similar to standard cameras, the depth camera captures two-dimensional data for a plurality of pixels containing a video image. These values are color values for the pixels, generally red, green, and blue (RGB) values for each pixel. In this way, objects captured by the camera appear as two-dimensional objects on the monitor. However, unlike a conventional camera, a depth camera also captures z-elements of the scene that represent depth values for the scene. Because depth values are typically assigned to the z-axis, depth values are often referred to as z-values.

In operation, a z-value is captured for each pixel of the scene. Each z-value represents a distance from a camera to a particular object in the scene corresponding to the associated pixel. In addition, the maximum detection range is defined as the range in which depth values begin not to be detected. This maximum range plane can be used by embodiments of the present invention to provide user defined object tracking. Therefore, using a depth camera, each object can be tracked in three dimensions. As a result, the computer system of embodiments of the present invention may use z-values along the two-dimensional pixel data to create an improved three-dimensional interaction environment for the user. More information on depth analysis is obtained with reference to US patent application Ser. No. 10 / 448,614, filed May 29, 2003, systems and methods for providing a real-time three-dimensional interaction environment.

Although a depth camera is used in accordance with one embodiment, it should not be construed as necessary to identify the position and coordinates of an object in three-dimensional space. For example, in the scenario shown in FIG. 2, the distance between the object 215 and the camera 105 can be inferred by measuring the distance of the leftmost and rightmost LEDs 212. As the LEDs 212 get closer to the image generated by the image capture device 105, the object 215 is further away from the camera 105. Therefore, reasonably accurate estimation of z-axis coordinates can be made from two-dimensional images generated by a typical digital camera.

According to FIG. 3, the input image processor 302 translates the captured video images (which may be depth images) of the control object into signals delivered to the output image processor. In one embodiment, the input image processor 302 is programmed to separate the control object from the background in the captured video image via depth information and generate an output signal in response to the position and / or movement of the control object. The output image processor 304 is programmed to influence translational and / or rotational movement of an object on the video display device 306 in response to signals received from the input image processor 302.

These additional images of the present invention may be implemented by one or more processors executing software instructions. According to one embodiment of the invention, a single processor executes both input image processing and output image processing. However, as shown in the figures, processing operations are shown as being split between the input image processor 302 and the output image processor 304. The invention should not be construed as limited to any particular processor configuration, such as two or more processors. The plurality of processing blocks shown in FIG. 3 are shown for convenience of description only.

4 is a schematic block diagram of a computer processing system configured to implement embodiments of the present invention described herein. The processing system may represent a computer-based entertainment system implementation that includes a central processing unit (CPU) 424 and a graphical processing unit (GPU) 426 coupled to the main memory 420. have. The CPU 424 is also coupled to an input / output processor bus 428. In one embodiment, GPU 426 includes an internal buffer for fast processing of pixels based on graphics data. In addition, GPU 426 output processing to convert the processed image data into standard television signals such as, for example, NTSC or PAL, for transmission to a display device 427 connected external to the entertainment system or its elements. It can include parts or functions. Optionally, the data output signals may be provided to a display device other than a television monitor, such as a computer monitor, liquid crystal display (LCD) device, or other type of display device.

IOP bus 428 connects CPU 424 to various input / output devices and other buses or devices. IOP bus 428 may include input / output processor memory 430, controller 432, memory card 434, universal serial bus port 436, (also known as a Firewire interface). Is connected to the IEEE1394 port 438 and the bus 450. The bus 450 includes an operating system (OS) ROM 440, a flash memory 442, a sound processing unit (SPU) 444, an optical disk control unit 446, and a hard disk. Several other system components, including drives (HDDs) 448, connect to the CPU 424. In one aspect of this embodiment, the video capture device may be directly connected to the IOP bus 428 for transmission to the CPU 424; The data from the video capture device can be used to change or update the values used to generate graphical images on the GPU 426. In addition, embodiments of the present invention may utilize various image processing configurations and techniques, such as those described in US patent application Ser. No. 10 / 365,120, filed Feb. 11, 2003, methods and apparatus for real time motion capture. Can be. The computer processing system can run in a CELL ^™ processor.

5 is a block diagram of the configuration of elements of a video game console suitable for use with an manipulated object functioning as an optional input device in accordance with one embodiment of the present invention. Exemplary game console 510 includes a multiprocessor unit (MPU) 512 for overall control of console 510, main memory 514 used for storage of various program operations and data, and geometry processing. A vector computing unit 516 for performing the necessary floating point vector operations, an image processor 520 for generating data and outputting video signals to the monitor 110 based on the controls from the MPU 512, an MPU ( 512 or a graphics interface (GIF) 522 for performing adjustments and the like over the transmission bus between the vector computation unit 516 and the image processor 520, transferring data to and from the peripherals. Input / output port 524 for receiving data, for example, an internal OSD functional ROM (OSDROM) configured by flash memory to perform control of the kernel, and calendar and other It is equipped with the (528 RTC;; real time clock) real time clock having Murray function.

Main memory 514, vector computation unit 516, GIF 522, OSDROM 526, real time clock 528, and input / output port 524 to MPU 512 across data bus 530. It is connected. In addition, the bus 530 is connected to an image processing unit 538, which is a processor for expanding compressed video and texture images, to develop image data. For example, the image processing unit 538 can decode and develop bitstreams, macroblock decoding, and inverse discrete cosine transforms in accordance with MPEG2 or MPEG4 standard formats. perform functions for discrete cosine transformations, color space transformation, vector quantization, and so on.

The acoustic system includes a sound processing unit (SPU) 571 for generating music or other sound effects based on instructions from the MPU 512, and an acoustic buffer 573 in which waveform data can be recorded by the SPU 571. ), And a speaker 575 for outputting music or other sound effects generated by the SPU 571. Speaker 575 may be embedded as part of monitor 110 or may be provided as a separate audio line-out connection attached to external speaker 575.

A communication interface 540 having functions of input / output of digital data and which is an interface for inputting digital content according to the present invention is provided connected to the bus 530. For example, via communication interface 540, user input data may be sent to a server terminal in a network to accommodate online video game applications, and status data may be received from the server terminal. An input device 532 (also known as a controller) for input of data (e.g., as key input data or coordinate data) with respect to the console 510, and various programs and data (i.e. objects Is connected to an input / output port 524 for reproducing the contents of the optical disk 569, such as a CD-ROM having stored therein data, texture data, and the like.

The invention further includes a digital video camera 105 coupled to the input / output port 524. Input / output port 524 may be implemented by one or more input interfaces, including serial and USB interfaces, and digital video camera 190 may be a USB input or other conventional suitable for use with camera 105. It is preferable to use an interface.

The image processor 520 described above includes a rendering engine 570, an interface 572, an image memory 574, and a display control device 576 (eg, a programmable CRT controller, etc.). The rendering engine 570 executes operations for rendering the predetermined image data in the image memory through the memory interface 572 in response to the rendering instructions supplied from the MPU 512. The rendering engine 570 may, for example, display more than 320x240 pixels or 640x480 pixels of image data in real time, in particular more than tens to tens of tens of bundles per second, for example NTSC or PAL. Has the ability to render at high speeds. The bus 578 is connected between the memory interface 572 and the rendering engine 570, and the second bus 580 is connected between the memory interface 572 and the image memory 574. The first bus 578 and the second bus 580 each have a bit width of, for example, 128 bits, and the rendering engine 570 may execute high speed rendering processing in relation to image memory. The image memory 574 uses an integrated memory structure, for example, where the texture rendering area and the display rendering area can be set to uniform areas.

The display controller 576 stores the texture data retrieved from the optical disk 569 through the optical disk device 536 or the texture data generated in the main memory 514 via the memory interface 572 through the image memory 574. To the texture rendering area. The image data rendered in the display rendering area of the image memory 174 is read through the memory interface 572, output to the monitor 110, and displayed on the screen.

FIG. 6 is a block diagram illustrating functional blocks used to track and identify a group of pixels corresponding to a user input device as manipulated by a user in accordance with one embodiment of the present invention. The functions shown by the blocks are performed by software executed by the MPU 512 in the game console 510 of FIG. 5. In addition, not all of the functions illustrated by the blocks in FIG. 6 are used for each embodiment.

Initially, pixel data input from the camera is supplied to the game console 510 via the input / output port interface 524, enabling the following processes to be performed at the game console. First, as each pixel of the image is sampled, for example, in a raster structure, a color segmentation processing step S201 is performed, in which the color of each pixel is determined and the image is a variety of different colors. It is divided into two-dimensional segments. Next, for certain embodiments, a color shift positioning step S203 is performed, in which regions in which segments of other colors are adjacent are more specifically determined, thereby defining the position of the image where distinct color shifts occur. . Geometry processing step S205 is then performed, depending on the embodiment, straight lines, which are mathematical or geometric terms, corresponding to edges of the object of interest, including performing an edge detection process or computation on area statistics. Define curves and / or polygons.

The three-dimensional position and orientation of the object is computed in step S207, in accordance with an algorithm to be described in connection with the following detailed description of preferred embodiments of the present invention. Data in three-dimensional position and orientation also undergoes a processing step S209 for Kalman filtering to improve performance. This process is performed to evaluate where the object will be over time, to reject spurious measurements that are not possible, and are therefore considered to be outside of the truth data set. Another reason for performing Kalman filtering is that while the exemplary display runs at 60 Hz, the camera 105 generates images at 30 Hz, so Kalman filtering is used to determine the intervals of data used to control the operation of the game program. Fill it. Flattening of discrete data via Kalman filtering is well known in the computer art and will not be described any further.

FIG. 7 shows a schematic block diagram of an example image processing system 700 for mapping the movements of an object 705 in a spatial volume 702 seen by the image capture device 105. In this embodiment, as the object 705 is moved by distance x ₁ in the three-dimensional space 702, the image processing system 700 interprets the captured video images of the object 705, and the object 705 Identify the movement of the control panel and generate an operation substantially corresponding to the display screen 110.

Specifically, the image capture device 105 includes a digital image sensor for generating image data representing an image formed by light that affects the digital image sensor after passing through the lens as is known in the art. Image capture device 105 may also include an analog video camera that generates an analog signal representing an image formed by light. In the latter case, the analog signal is converted into a digital representation of the image prior to processing by the recognizer 710. Image data representing successive two-dimensional images of the three-dimensional space 702 is transferred to the recognizer 710. The recognizer 710 may perform various processing steps as described above with reference to FIG. 6 to identify the object 705. The position of the object 705 is communicated to a mapper 712. For example, the absolute coordinates of the object 705 in the three dimensional space 702 can be computed and passed to the mapper 712. Coordinates along the x and y axes can be determined from the position of the object as represented in each image. The coordinates of the object 705 along the z-axis can be inferred from the size of the object. That is, the closer the object 705 is to the image capture device 105, the larger will appear in the image. Therefore, the measurement of the diameter of the object appearing in the image can be used to calculate the distance from the image capture device 105, ie the z-axis coordinates.

In addition to the location information, recognizer 710 may identify commands received from object 705. The instructions may be interpreted from the transmissions / deformations of the object 705, sound and light generation, and the like. Commands received from the object 705 may be interpreted by the recognizer, and data corresponding to the received commands may be passed to the application 714. The application 714 may be a game application or other computer application that receives user input from the image capture device 105. In one embodiment, mapper 712 inputs absolute coordinates from recognizer 710 and maps these coordinates to scaled output coordinates according to the amount of gearing. In another embodiment, the mapper 712 receives continuous coordinate information from the recognizer 710 and converts changes in the coordinate information into vector movements of the object 705. For example, if the object 705 is moved by a time t by the distance t ₂ from ₁ x _1, the vector (x _1, 0, 0) is generated and transmitted to the application 714. Time t ₁ to time t ₂ may be a time interval between successive frames of video generated by image capture device 105. The mapper 712 can scale the vector according to a scaling algorithm, for example, by multiplication of the vector by the gearing amount G. In another embodiment, each of the coordinates is, for gearing element, for corresponding, is multiplied by a G _x, G _y and G _z. Thus, the corresponding movement of the virtual object 705 ′ as shown within the display 110 may be a distance x _{2 that is} different from x ₁ .

The application 714 can change the gearing amount in accordance with instructions 713 received from the recognizer 710 or in accordance with the normal operation of the software sending the gearing data to the mapper 712 to cause the gearing amount to change. . Gearing data may be sent to the mapper 712 in response to user commands, various events, or modes of operation of the application 714. Therefore, the amount of gearing can be changed in real time as in response to user commands or as controlled by software. Therefore, the mapper 712 sends to the application 714 the output vector of the movement of the object 705, the output that changes with respect to the change in position of the object 705 in the space 702, and the amount of gearing. The application 714, which is a video game in one embodiment, translates the output vector into a corresponding action to be displayed on the display 110.

8 illustrates an example application for the image processing system 700 of FIG. 7. Computer system 102 includes an image capture device 105 showing a scene 810. Scene 810 includes user 802 holding object 804 recognized by recognizer 710 (FIG. 7). The application program, which is a checker game in this embodiment, recognizes the instructions distributed from the object 804 to pick or place the checkers 808 on the checkerboard 801. As the user moves the object 805 in front of the image capture device 105, the computer 102 processes the images to identify the movement of the object 804, which is then translated into a virtual object 805 ′ on the display 110. To translate). The distance that the virtual object 805 'travels in relation to the real object 805 depends on the amount of gearing 806. In this embodiment, the gearing amount is represented by "3" on the display 110. In one embodiment, the amount of gearing is user selectable. The larger the amount of gearing, the smaller the movement of the real object 805 is needed to affect the specific distance movement of the checker 805 'on the display 110.

9 and 10 show an example controller 900 that interacts with the image capture device 105 (FIG. 1). Controller 900 includes an interface 902 that includes a plurality of interface devices including various buttons and joysticks. The controllers discussed herein can be wired or wireless. Technologies such as WiFi, Bluetooth, IR, sound, and lights can operate to interface with a computer, such as a game console. In one embodiment, the controller 900 has an LED arrangement 905. The LED arrangement may consist of various layouts, each of which includes a 2x2 stack located at the vertex of the virtual rectangular or square blinding box. By tracking the position and deformation of the blinding box projected on the image plane generated by the image capture device, the deformations can be analyzed in the video analyzer to decode the position and orientation information of the controller.

LED array 905 may generate infrared or visible light. Image capture device 105 (FIG. 1) may identify LED array 905 as described above with reference to various other progressive embodiments. Each controller may be designated from player 1 to player 4, for example, using a switch 910 that allows a user to select among player numbers 1-4 or any number of players. Each player number selection corresponds to a unique pattern or modulation of the LEDs illuminated by the LED array 905. For example, for player 1, the first, third and fifth LEDs are lit. This player information is encoded and transmitted in a repeating form over time over a plurality of video frames. It is desirable to take an interleave structure so that controller or device LEDs can be exchanged between tracking mode and transmission mode. In tracking mode, all the LEDs can be turned on during the first part of the cycle. In the transmission mode, the information can be modulated by the LEDs during the second part of the cycle. Over time, the LEDs transmit tracking and communication information to a video analyzer or a suitable device capable of receiving signals. In the transmit mode, the LEDs may encode information representing the player ID. The duration and duty cycle may be selected to adjust the tracking speed, emission conditions, number of controllers, and the like. By embedding communications and tracking information, the video capture device is supplied with appropriate information to calculate tracking parameters for each controller and identify between the controllers. This identification can be used in the video analyzer to isolate each physical controller when monitoring and tracking the position and direction of controller movement and other metrics.

In the transmission mode, other information, including commands or status information, may be transmitted by the controller or device LEDs in accordance with known encoding and modulation schemes. At the receiver side, a video analyzer connected to the video capture device is synchronized with and tracks the state of the LEDs and decodes information and controller movements. Higher bandwidth can be obtained by modulating the data over frames in the transmission mode cycle.

User interaction with the interface 902 may cause one or more of the LEDs in the LED arrangement 905 to modulate and / or change color. For example, as the user moves the joystick, the LEDs may change brightness or transmit information. Changes in density or color can be monitored by a computer system and provided to the game program as a density value. In addition, each button can be mapped to a color change or intensity of one or more of the LEDs in the LED array 905.

As the controller 900 is moved in three-dimensional space and rotated in one of the roll, yaw, or pitch directions, the image capture device 105, along with the computer system 102, can identify these changes, and the image plane A two-dimensional vector for describing the movement of the image or a three-dimensional vector for describing the movement in the three-dimensional space can be generated. The vector may be provided as a series of coordinates describing the relative movement and / or as an absolute position relative to the image capture device 105. As will be apparent to those skilled in the art, the movement of the controller closer to the image capture device 105 can be identified by the LED arrangement appearing to spread outward, while the plane of the plane (image plane) perpendicular to the line of sight of the image capture device 105 can be identified. Movement can be identified by an absolute position within the image capture area.

The rectangular configuration of the LEDs 905 allows the movement of the controller 900 on three axes and the rotation about each axis to be detected. Although only four LEDs are shown, this is for illustrative purposes only, and any number of LEDs in any configuration is possible. As the controller 900 is thrown forwards or backwards, the top LED and the bottom LED will be closer together, and the left and right LEDs will be keeping the same distance. Similarly, as the controller yaws to the left or to the right, the top LED and the bottom LED are keeping the same distance from each other, but the left LED and the right LED will appear to approach. The rolling movement of the controller can be detected by identifying the direction of the LEDs on the image plane. As the controller moves closer to the image capture device 105 along the line of sight, all of the LEDs will appear closer to each other. Finally, the movement of the controller along the image plane can be tracked by identifying the location of the LEDs on the image plane, thereby identifying the movement along the respective x and y axes.

The controller 900 may also include a speaker 915 for producing an audible or ultrasonic sound. Speaker 915 may generate sound effects for increased interaction and may transmit instructions distributed from interface 902 to a computer system having a microphone or other elements for receiving transmissions.

11 illustrates an example application for the controller of FIGS. 9 and 10. In this application, driving simulation interprets the rotation of the controller 900 as the rotation of the steering wheel of the virtual vehicle. As the user (not shown) rotates the controller 900 as indicated by arrow 1105, virtual steering wheel 900 ′ is rotated on display 110 as indicated by arrow 1105 ′. In one embodiment, the amount of gearing indicative of the amount of rotation of the steering wheel 900 'for each angle of rotation of the controller 900 is user selectable as described above with reference to FIG. In yet another embodiment, the amount of gearing is controlled by software as represented in the example graph 1200 shown in FIG. 12. In this embodiment, the amount of gearing changes in relation to the distance from the center of the controller 900, ie the virtual direction. This allows for 540 ° rotation of the virtual steering wheel 900 ′ with only 90 ° rotation of the controller 900. By maintaining a low amount of gearing near the 0 ° (center) position, high control can be maintained during high speed driving, which typically does not require significant steering rotation. As the controller 900 is rotated further away from the center position, the amount of gearing increases as shown in the graph 1200, generally to accommodate the sharper rotation required at slower speeds.

13 shows another example controller 1300 having a steering wheel 1305 that can be operated by a user. In this case, the rotations of the steering wheel 1305 and the operation of the buttons of the interface 1302 are interpreted by the controller 1300 making data transfer via the LEDs 1310.

14 illustrates an example application of the controller 1300. In this embodiment, the application is a driving simulation that receives commands distributed in response to the rotation of the steering wheel 1305 and translates these commands as corresponding rotations of the virtual steering wheel 1305 in the display 110. The amount of gearing that scales the rotation of the steering wheel 1305 to the corresponding rotation of the virtual steering wheel 1305 'may vary in response to user interaction with the interface 1302 (FIG. 13) or in response to software control. FIG. 15 shows an example graph 1500 showing example changes in the amount of gearing over time. In one embodiment, changes in gearing can be dynamically set by the user during game play. In addition, variations of gearing may be flat, abrupt, gradual, long, short, or a combination thereof, as shown by graph 1500. Therefore, the gearing can be set or changed by the game over time during the game session to provide a more realistic interactive experience. In addition, as the user is able to control gearing, another dimension of control over the predetermined controls found in conventional games is possible.

16 illustrates another example application of an image processing system corresponding to user interaction. In this embodiment, the user 1602 interacts with the baseball simulation by swinging a toy baseball bat 1605 that can be recognized as an input object by the image processing system. As the toy baseball bat swings, the user and / or software can control the amount of gearing to manipulate the speed or distance of the virtual baseball bat 1605 '. In one embodiment, the bat may include a number of buttons that can be pressed during game play, and the gearing may change as the buttons are pressed. In yet another embodiment, the user can preset or program a combination of gearing levels to be applied when the bat is swinging.

FIG. 17 shows a graph 1700 showing an example amount of gearing at different times t ₁ to t ₅ depending on the length of the swing. These gearing quantities may be set by the user before swinging the bat or may be dynamically set by the game in response to the bat's position as seen by the camera 105. Also shown is how gearing can change over time, be constant or increase in specific intervals. In graph 1700, gearing is set higher at times t _2- t ₃ so that the swing of the bat becomes more powerful when the contact is made with the ball, and is relaxed at times t _3- t _{4 when} the bat is already in contact. have. In one embodiment, such other times may be evaluated by the user or determined by a computer.

In one embodiment, the user may make several practice swings, after which the computer may map multiple example time slots corresponding to the user's actual swing ability. Thereafter, the user can assign specific gearing to each time interval, depending on how the user wants to affect game interaction. Once gearing is established, movement of the bat 1605 of the user may be mapped to movement of the bat 1605 '(e.g., a game object). In addition, gearing may be preset by the game during other operations, set by the user during game play, and adjusted in real time during game play.

18 illustrates another example application of an image processing system in response to user interaction. In this embodiment, the user (not shown) points to the release of the football by making a drawing movement with the toy football and pressing the actuator and interacts with the football simulation. Of course, instead of toy football, a controller can be used. The virtual player 1802 manipulates the virtual football 1805 in response to user interaction. In one embodiment, the actuator causes the LED to light up or change the color perceived by the image processing system as a command sent to the football simulation application. After releasing the ball, the user can control certain actions on the field, such as the movement of the selected receiver. In one embodiment, the release of the ball causes the application to send gearing data to the mapper (FIG. 7) to change the amount of gearing, for example to enable identifying better movements of the input object. do.

19 shows an example graph 1900 showing a change in the amount of gearing upon "release" of football. Although shown as a simple graph 1900, gearing can take any form depending on the game object.

20 shows a flowchart 2000 illustrating an example process for receiving user input for a computer program. The process begins as indicated by start block 2002 and proceeds to operation 2004 where the location of the control input is identified. The control input may be a position of an input object in the three-dimensional space described above with reference to FIG. 7, or a position of the user interface device described above with reference to FIG. 14. The position may be represented by a representative value, or a plurality of values such as a vector. After identifying the location of the control input, the process proceeds to operation 2006.

In operation 2006, a determination is made whether the location has changed. If the position has not changed, the process returns to operation 2004. In one embodiment, operation 2004 is postponed until a new frame of image data is received from the image capture device. In operation 2006, if it is determined that the position has changed, the process proceeds to operation 2008.

In operation 2008, the motion vector is computed. The motion vector can have any number of dimensions. For example, if an input object in three-dimensional space is in motion, the motion vector can describe the motion as a three-dimensional vector. However, if the movement is a one-dimensional control input such as a steering wheel, then the movement vector is a one-dimensional vector describing the amount of rotation of the steering wheel. After determining the motion vector, the process proceeds to operation 2010.

In operation 2010, the motion vector is multiplied by the current gearing quantity to determine the input vector. The current gearing quantity can be a scalar quantity or a multidimensional value. If it is a scalar amount, all dimensions of the motion vector are multiplied by the same amount. If the amount of gearing is multidimensional, then each dimension of the motion vector is multiplied by the amount of gearing of the corresponding dimension. The amount of gearing can change in response to user input and can be controlled by software. Therefore, the current gearing amount can change from moment to moment. After multiplying the current vector by the amount of gearing, the process proceeds to operation 2012.

In operation 2012, a new position of the virtual object is computed using the input vector computed in operation 2010. The new position may be a camera position or an object position such as a virtual steering wheel. The virtual object may not be displayed on the display screen. When the new position of the virtual object is calculated, the process proceeds to operation 2014 in which data representing the new position is transferred to the application program. The process then ends as indicated by end block 2016. However, this process is exemplary in nature, and other options are possible.

In one embodiment, the operations will include detection of the input device and movement of the input device. The movement of the input device is determined by the camera looking at the input device. The gearing value is applied by user control or by preset or pre-programmed settings. If set by the user, the gearing value can be selected, for example, by allowing the user to hit a button on an input device (eg, a controller). Depending on the amount of gearing, the movement control is mapped to the object of the computer game. If the user uses the input device to control the motion picture of the game, the gearing that is set or controlled to be set will affect how movement by the input device is mapped to motion by the motion picture of the computer game. Therefore, the gearing and the change in the gearing allow for the dynamic application of the mapped response by an object that may be part of a computer game.

In various embodiments, the image processing functions described above to determine the concentration value, controller player number, direction and / or location of one or more input objects including the controllers are performed in a process executed in a computer system. . The computing system may also request or accept data generated from video or audio processing, such as data including a controller player number, the direction and / or location of one or more input objects including controllers, controller operation, and the like. It executes a main process called an application program which may be a game application. In various embodiments, the process of performing video and / or audio processing functions is a driver for a video camera or video / audio surveillance device, the driver being generally any of the embodiments known in the art. Provides data to the main process through a form of interprocess communication. By executing the main process which is game software or other application program, the process of performing the video or audio processing is executed in the same processor or another processor. For example, using a procedure call, it is possible to have a common process for both video or audio processing and game functions in the same process. Therefore, while an input vector or other information is described as being provided as a "program", the present invention uses a procedure call or other software function to enable a single process to perform both image processing and game functions. This includes providing this data to a routine. Accordingly, one or more processes running on a conventional processor core or multiple processor cores perform video and / or audio processing as described herein, and the separate process performs game functions.

The invention is intended to be used as other user input mechanisms, mechanisms for tracking the angular direction of sound and / or mechanisms for actively or passively tracking the position of an object, mechanisms using machine vision, combinations thereof The tracked object may include auxiliary controls or buttons to manipulate feedback to the system, which feedback may affect transmission or modulation, encode status, and / or commands from the device being tracked. Or but not limited to, buttons, pressure pads, etc., as well as light emitting from a light source, sound modifying means, or other suitable transmitters and modulators that transmit to the device.

The invention includes game consoles, game computers or computing devices, hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, main frame computers, and the like. Other computer system configurations. The invention may be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a network. For example, online game systems and software may also be used.

Although the above embodiments have been described, the present invention may utilize various computer-implemented operations including data stored in computer systems. These operations are those requiring physical manipulations of physical quantities. Typically, but not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated. In addition, the operations performed are often referred to in terms such as reproduction, identification, determination or comparison.

The operations described herein, which form part of the invention, are useful machine operations. The invention also relates to an apparatus for performing these operations. The apparatus may be specially configured for the necessary purposes, such as the carrier network discussed above, or it may be a general purpose computer which is selectively operated or configured by a computer program stored in the computer. In particular, various general purpose machines may be used using computer programs recorded in accordance with the teachings of the present disclosure, or it may be more convenient to construct a more specialized apparatus to perform the necessary operations.

The invention may also be embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data that can be read by a computer system. Embodiments of computer readable media may include hard drives, network attached storage (NAS), read-only memory, random-access memory, flash based memory. , CD-ROMs, CD-Rs, CD-RWs, DVDs, magnetic tapes, and other optical and non-optical data storage devices. The computer readable medium may be distributed over computer systems connected to a network such that the computer readable code is stored and executed in a distributed form.

Also, although gearing has been discussed in connection with video games, it should be understood that gearing can be applied to any computer controlled environment. In one embodiment, gearing may be associated with a computer input device that enables the interaction, selection or input of information. Applying different gearing during different input or interaction operations may result in a more advanced operation that is not typically found in environments with preconfigured control settings. Thus, as defined herein, embodiments of gearing should be given a wide range of applications.

Once gearing is determined, gearing may be applied to the operation transferred to the computer program. As mentioned above, the tracking of the operation of the input device may be achieved through image analysis, inertial analysis, and audible analysis. Embodiments of operations include throwing an object, such as a ball, swinging an object, such as a bat or golf club, pumping a hand pump, opening or closing a door or window, Turning the steering wheel or other vehicle control, martial arts movements such as punches, sanding movements, waxing or peeling off, painting the house, shaking, rattles, rolling, playing football Any vector such as, throwing a baseball, turning a handle, 3D / 2D mouse movement, scrolling movement, known contoured movement, any recordable movement, pumping tires in any direction in space Accurately based on user manipulations that can be recorded, tracked, and repeated within motions along the front and back, along paths, noise floors, splines, etc. It comprises a movement having a support time and the start time, not limited to this. Each of these operations may be pre-recorded from the route data and stored as a time-based model. Therefore, gearing may be applied to one of these operations, depending on the degree of gearing set by the user or program.

Although the foregoing invention has been described in detail for clarity of understanding, it is clear that certain changes and modifications may be made within the scope of the appended claims. Accordingly, the present embodiments are illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.

Claims (21)

A method for interacting with a computer game system comprising a video capture device for capturing image data, the method comprising: Identifying an input device; Defining an action that is to be executed by the computer game system and that is responsive to interaction of the input device; Establishing gearing between the operation and the interaction of the input device; And Adjusting the gearing while the operation is performed, And said gearing establishes a ratio between the interaction of said input device and said actions. The apparatus of claim 1, wherein the input device comprises: (i) light moving to define the interaction, (ii) an inertial sensor changing to define the interaction, and (iii) to define the interaction A changing sound device, or (iv) a mixer receiving the changes defining the interaction. The method of claim 1, wherein the movement of lights is detected within the volume space captured by the video capture device. The method of claim 3, wherein Continuously analyzing the sensed movement of the lights; And And causing the update of the operation to be made based on the sensed movement or on a program basis. 3. The method of claim 2, wherein the light is a light emitting diode. 6. The method of claim 5, wherein the state of light is represented by a setting of on / off combinations. The method of claim 1, wherein the operation comprises: (a) setting the amount of movement by the object of the computer game when executed by the computer game, (b) setting a variable for executing the process, and (c) setting the rate of change into the process that affects sound or vibration, or (d) setting the rate of change in the movement of the graphical object. Method for interactive interface. 3. The method of claim 2, further comprising: at least periodically analyzing data from the inertial sensor, the acoustic sensor, or a mixer; And reapplying the operation at least periodically with changes updated by the gearing based on the periodic analysis of the data. 3. The method according to claim 2, wherein when the gearing is adjusted, the adjustment is applied (a) step by step, (b) gradually or (c) smoothly to affect the operation. And a method for interacting with a computer game system. The method of claim 2, wherein the data from the inertial sensor is analyzed to identify coordinate information of the input device, and the coordinate information is converted by mapping into vector data applied to the operation. A method for interacting with a computer game system. 11. The method of claim 10, wherein the vector data is scaled in response to adjustments of the gearing. A system for performing dynamic user interaction between user actions and actions to be performed by a computer program running on a computing system, the system comprising: The system includes an input device for interfacing with a computer program to be executed by the computing system, The input device has gearing control for establishing a scaled value between movement data of the input device as mapped to an operation to be applied by the computer program, And the gearing is adjusted while the computer program is running. 13. The dynamic user interaction of claim 12, wherein the gearing can be set prior to the interaction with the computer program, can be set during the interaction, or can be set after the interaction. System for performing the operation. 13. The device of claim 12, wherein the input device is a controller or hand-held object comprising at least one of light emitting diodes, an inertial sensor, an acoustic device, or a mixer of inputs. System for performing dynamic user interaction. 15. The system of claim 14, wherein the acoustic device includes a speaker for conveying acoustic or ultrasonic signals causing the computing system to cause the operation. 15. The dynamic user interaction of claim 14, wherein the controller comprises a microphone for receiving input from a user to communicate with the computing system to cause operation by the computer program. System for performing the operation. 15. The dynamic according to claim 14, wherein the inertial data from the inertial sensor is analyzed to identify coordinate information of the input device, and the coordinate information is converted by the mapping into vector data applied in the operation. System for performing user interaction. 13. The dynamic user of claim 12 wherein the gearing may be set before an interaction with an object of the computer program, may be set during the interaction, or may be set after the interaction. System for performing interactions. 13. The method of claim 12, wherein the computer program is an interactive application, the interactive application being one of a video game, a program, an internet program, or an interfacing control program. System for performing user interaction. The method of claim 1, And adjusting the gearing while the operation is performed, adjusting the gearing according to at least one of a time interval, a rotation angle of the input device, and a button press of the input device. Method for interactive interface. The method of claim 12, And the gearing is adjusted during execution of the computer program according to at least one of a time interval, a rotation angle of the input device and a button press of the input device.

Images