CN102224476B - Multi-process interactive systems and method - Google Patents

Abstract

A kind of multi-process interactive systems are described.The system includes many processes run in processing.The separable program that process includes application program performs context so that each application program includes at least one process.The event of each process is translated to data capsule by the system.Data capsule includes initiating the expression unrelated with application program of the event data of the state of a process information of the content of data capsule and event.Data-message is sent in pond or reservoir by the system.Each process as identification process operation, wherein, the identification process recognize in pond including the data capsule with recognizing the interactive function of process and/or the corresponding content of mark of identification process.The data capsule of identification process retrieval from pond, and perform the processing to the content of the data capsule of identification suitably.

Description

Multi-process interaction system and method

related application

This application claims the benefit of United States (US) Patent Application Serial No. 61/105,243, filed October 14, 2008.

This application claims the benefit of US Patent Application Serial No. 61/105,253, filed October 14, 2008.

This application is a continuation-in-part of US Patent Application Serial No. 12/109,263, filed April 24, 2008.

This application is a continuation-in-part of US Patent Application Serial No. 12/417,252, filed April 2, 2009.

This application is a continuation-in-part of US Patent Application Serial No. 12/487,623, filed June 18, 2009.

This application is a continuation-in-part of US Patent Application Serial No. 12/553,845, filed September 3, 2009.

This application is a continuation-in-part of US Patent Application Serial No. 12/557,464, filed September 10, 2009.

This application is a continuation-in-part of US Patent Application Serial No. 12/572,689, filed October 2, 2009, which is a continuation-in-part of US Patent Application Serial No. 7,598,942.

technical field

The present invention relates to embodiments of the representation, manipulation and exchange of data within and between computing processes.

Background technique

Conventional programming environments do not fully support multiple computer processing units (CPUs) and cross-network execution, or flexible sharing of data among numerous computational processes. User-facing computer programs have traditionally been structured such that most and all graphical output of the process is produced by a single computing process. This mechanism, while standard and well supported by toolchains, development environments, and operating systems, is poorly scaled and a clear contributor to the bloat and fragility of widely used contemporary applications.

merge by reference

Each patent, patent application, and/or publication mentioned in this specification is hereby incorporated by reference in its entirety as if each individual patent, patent application, and/or publication were specifically and individually indicated to be incorporated by reference And to combine.

Description of drawings

Figure 1A is a block diagram of a multi-process interactive system, according to an embodiment.

Figure IB is a block diagram of a multi-process interactive system according to an alternative embodiment.

Figure 1C is a block diagram of a multi-process interactive system according to another alternative embodiment.

Figure 2 is a flowchart of the operation of the multi-process interactive system according to an embodiment.

3 is a block diagram of a processing environment including data representation using slawx, protein, and pool, according to an embodiment.

Figure 4 is a block diagram of a protein according to an embodiment.

Fig. 5 is a block diagram of a description according to an embodiment.

Figure 6 is a block diagram of an ingest, under an embodiment.

7 is a block diagram of a salad, under an embodiment.

Figure 8A is a block diagram of proteins in a pool, according to an embodiment.

Figure 8B illustrates a salad header format, according to an embodiment.

Figure 8C is a flow diagram for using proteins, under an embodiment.

Figure 8D is a flowchart for constructing or generating proteins, according to an embodiment.

9 is a block diagram of a processing environment including data exchange using slaws, proteins, and pools, under an embodiment.

10 is a block diagram of a processing environment including multiple devices and a number of programs running on one or more of the devices in which plasma constructs (e.g., pools, proteins, and salads) according to an embodiment ) are used to allow a large number of running programs to share and collectively respond to events generated by the device.

11 is a block diagram of a processing environment including a plurality of devices and a number of programs running on one or more of the devices in which plasma constructs (e.g., pools, proteins, and salads) according to an alternative embodiment Used to allow a large number of running programs to share and collectively respond to events generated by a device.

12 is a block diagram of a processing environment including multiple input devices coupled among a multitude of programs running on one or more devices in which plasma constructs (e.g., pools, protein and Salad) are used to allow a large number of running programs to share and collectively respond to events generated by input devices.

13 is a block diagram of a processing environment including multiple devices coupled among a multitude of programs running on one or more devices in which plasma constructs (e.g., pools, proteins, and Salad) is used to allow a large number of running programs to share and collectively respond to device-generated graphics events.

14 is a block diagram of a processing environment including multiple devices coupled among a multitude of programs running on one or more devices in which plasma constructs (e.g., pools, proteins, and Salad) is used to allow status inspection, visualization and debugging of running programs.

15 is a block diagram of a processing environment including multiple devices coupled among a multitude of programs running on one or more devices in which plasma constructs (e.g., pools, proteins, and slaw) are used to allow influencing or controlling the properties of the state information generated and placed in the process pool.

Figure 16 is a block diagram of a gestural control system, under an embodiment.

Figure 17 is a diagram of a marker label, according to an embodiment.

Figure 18 is a diagram of gestures in a gesture vocabulary, under an embodiment.

Figure 19 is a diagram of orientations in a gesture vocabulary, under an embodiment.

Figure 20 is a diagram of two hand combinations in a gesture vocabulary, under an embodiment.

Figure 21 is a diagram of orientation blending in a gesture vocabulary, under an embodiment.

Figure 22 is a flowchart of gesture control, under an embodiment.

Figure 23 is an example of a command under an embodiment.

Figure 24 is a block diagram of a spatial operating environment (SOE) implemented by a multi-process interactive system, under an embodiment.

25 is a flowchart of the operation of a multi-process interaction system using input from a gesture control system, under an embodiment.

detailed description

Embodiments described herein include systems and methods for coordinating the behavior and output of multiple computer processes to produce interactive applications. Embodiments described herein are generally referred to as multi-process interactive systems, programs, or applications, including applications that are divided into multiple distinct computer processes that can execute in parallel. The collection of these processes can produce the portion of the overall system output through which the user interacts. A collection of these processes has access to a structured and well-defined data exchange mechanism for coordinating activities. The collection of these processes is operable to consume user input (eg, raw user input, weighted transformed user input, raw and weighted transformed user input, etc.) via a structured data exchange mechanism.

Embodiments described herein provide modularity of application components across computing process boundaries. As a result of the modularity provided, the embodiments described herein provide for component reuse, greater opportunity for interoperability, easier testing and verification, increased robustness, and fault tolerance.

In addition, current computers often contain multiple processor elements (eg, CPU cores). Embodiments herein scale much better on multiprocessor architectures than conventional application construction techniques. As a trend in computer design, this "multi-core" scaling is becoming more and more important, and manufacturing is increasingly catering to increased core counts rather than increased clock speeds.

Embodiments described herein enable dynamic construction, destruction and reorganization of processing components. Embodiments described herein enable the extension of structured data exchange mechanisms across multiple computers using networking (or other interconnection) protocols. Embodiments described herein enable dynamic transfer of processing components between computers. Embodiments described herein enable dynamic optimization of structured data exchange mechanisms according to the number, composition and execution context of participating processes. Embodiments described herein enable graphical output created on multiple computers to be combined on a single display. Embodiments described herein implement a shared, coordinated graphics context that includes multiple displays. Embodiments described herein implement a shared, coordinated multi-display graphics context comprising multiple displays driven by multiple computers. Embodiments described herein introduce automatic history buffer construction into the structured data exchange mechanism so that some amount of past data is always available to application components.

The following terms are intended to have the following ordinary meanings as they are used herein. The term "process" as used herein means a separable program execution context. Computer architectures and operating systems differ in the technical details of process implementation. The mechanisms described herein are configured to operate across a broad range of process implementations and facilitate hybrid application design or configuration that utilizes as many available computing resources as possible.

The term "device" as used herein means any processor-based device running one or more programs or algorithms, any processor-based device running under one or more programs or algorithms, and/or coupled or connected to a running One or more programs or algorithms and/or any device that is a processor-based device that operates under one or more programs or algorithms. As used herein, the term "event" means any event associated with the running or execution of a program or algorithm, a processor-based device, and/or a device coupled or connected to a processor-based device (for example, an event may include, but is not limited to input, output, control, state, change of state, action, data (regardless of the format of the data or the stage in processing with which the data is associated), etc.).

In the following description, numerous details are introduced to provide a thorough understanding of, and to enable explanation of, the embodiments described herein. However, one skilled in the art would understand that the embodiments may be practiced without one or more of these specific details, or with other components, systems, etc. In other instances, well-known structures or operations are not shown or described in detail to avoid obscuring aspects of the disclosed embodiments.

Embodiments herein include systems and methods that execute a multitude of processes on at least one processing device. The systems and methods of embodiments translate each process's events into data capsules. The systems and methods of embodiments transfer data capsules into multiple pools or storages. Each process operates as a recognition process that identifies in a pool a data capsule comprising content corresponding to an interaction function of the recognition process and/or an identity of the recognition process. The identification process retrieves identified data capsules from the pool and performs appropriate processing on the contents of the identified data capsules.

For example, FIG. 1A is a block diagram of a multi-process interaction system 10 according to an embodiment. The system 10 includes a processing device 11 hosting or executing any number of processes P1-P7. The example multi-process interactive system 10 includes or runs on one computer 11, but is not limited to one computer, and can run across any number and/or combination of processing devices or systems. The processes P1-P7 of the embodiment include detachable program execution contexts of one or more application programs, wherein each application program includes at least one process, but the embodiment is not limited thereto. Events generated or produced during the execution of each process are translated into some number of data capsules DC1-DC9, and the data capsules are transferred to a plurality of pools 15-17 or storages. The oval elements of system 10 represent pools 15-17, where a pool or reservoir is a mechanism for structured data exchange as described in detail below and in a related application. As described below, the data capsules DC1-DC9 (also referred to as data messages) passing through the pools 15-17 are generally described as "proteins".

Each process P1-P7 operates as a recognition process, wherein the recognition process identifies in the pool 15-17 a data capsule comprising content corresponding to the interaction function of the recognition process P1-P7 and/or the identity of the recognition process P1-P7. The recognition processes P1-P7 retrieve the recognized data capsules DC1-DC9 from the pool and execute the processes appropriate to the content of the recognized data capsules. The multi-process interactive system 10 is described in more detail below with reference to FIGS. 3-15 .

FIG. 1B is a block diagram of a multi-process interaction system 20 according to an alternative embodiment. The system 20 includes a processing device 21 hosting or executing any number of processes P1-PX, where X represents any number appropriate to the configuration of the processing device 21 and/or system 20 . System 20 also includes processing device 22 hosting or executing any number of processes P1 -PY, where Y represents any number appropriate to the configuration of processing device 22 and/or system 20 . The example multi-process interactive system 20 includes or operates across two processing devices 21/22, but is not limited to two devices, and may span any number and/or combination of processing devices or systems run. The processes P1-PX and P1-PY of the embodiment include detachable program execution contexts of one or more application programs, wherein each application program includes at least one process, but the embodiment is not limited thereto.

Events generated or produced during the execution of each process are translated into data capsules (not shown), and the data capsules are passed to one or more pools. The oval elements of system 20 represent pools, where a pool or repository is a mechanism for structured data exchange as described in detail below and in a related application. In this example, pool PL1 is hosted on processing device 21, but any number of pools may be hosted on processing device 21. Pools PL1-PLY are hosted on processing device 22, where Y represents any number appropriate to the configuration of processing device 22 and/or system 20, any number of pools may be hosted on processing device 22. System 20 also includes pools PL11-PLX, where X represents any number appropriate to the configuration of processing device 22 and/or system 20; any number of pools may be hosted in system 20. Any process and/or device that generates a data capsule can deliver a data capsule to any pool in the system.

Each process P1-PX/P1-PY operates as a recognition process, wherein the recognition process in the pool recognizes the interaction functions with the recognition process P1-PX/P1-PY and/or the recognition process P1-PX/P1-PY A data capsule identifying the corresponding content. The recognition process P1-PX/P1-PY retrieves the recognized data capsule from the pool and performs processing appropriate to the content of the recognized data capsule. The multi-process interactive system 20 is described in more detail below with reference to FIGS. 3-15 .

Embodiments herein include systems and methods that execute a multitude of processes on at least one processing device. A process of an embodiment includes separable program execution contexts for multiple applications, wherein each application includes at least one process. The systems and methods of embodiments translate events for each of a plurality of processes into data capsules. The data capsule includes an application-independent representation of the event data of the process that initiated the data capsule and the state information of the event. The systems and methods of embodiments transfer data capsules into multiple pools or storages. Each process of an embodiment operates as a recognition process. The recognition process identifies in the pool a data capsule comprising content corresponding to an interaction function of the recognition process and/or an identification of the recognition process. The identification process retrieves identified data capsules from the pool and performs appropriate processing on the contents of the identified data capsules.

Examples of embodiments described herein include systems and methods for coordinating the behavior and graphical output of multiple computer processes to enable interactive applications. Although this example is directed to graphics processes and graphics output, embodiments of the multi-process interactive system are not limited to graphics processes, and can be applied to any process running under any number of processing devices. A multi-process interactive system includes a portion of an application program divided into multiple distinct computer processes capable of executing in parallel, the aggregate of which produces an overall graphical output with which the user interacts. The set of processes has access to a structured and well-defined data exchange mechanism for coordinating activities such that the set of processes is operable to use user input via the structured data exchange mechanism.

As a more specific example, the following description teaches a multi-process graphics program, referred to herein as a square, as an illustrative example of an embodiment that coordinates the behavior and graphics output of multiple computer processes to produce an interactive application. The purpose of this illustrative example description is to show how the mechanisms disclosed herein operate at a level of detail sufficient to implement it for any interactive program. The mechanism (and indeed its component parts) is entirely generic and can actually be implemented in a variety of different ways. As is typical in such programs, the mechanisms disclosed herein provide primary services including, but not limited to, access to user input, fine-grained coordination of program state across processes, and coordination of graphical output.

The square program presented here serves to demonstrate several basic coordinations that are useful in real-world programs. The square program provides a flexible number of colored translucent squares to render on one or more computer displays. Each of these squares is implemented in a single computational process. The state and graphical details of each square depend on various factors, including user input actions, the state of other squares, and external messages passed globally. The square can be moved around on the display using an input device (eg mouse, touch screen, etc.). The gestural/spatial input system described in the related application can also be used to move the square, in which case the square can be located on any display available to the computer that is joining the gestural/spatial network.

FIG. 1C is a block diagram of a multi-process interaction system 100 according to another alternative embodiment. The system 100 includes processes and interconnections that combine to form example operations of a program. Solid rectangular elements (eg, generally elements M, P, S, G) represent processes in system 100 . Elliptical elements (eg elements Ui, Coo, Frame) represent pools, mechanisms for structured data exchange as described in detail below and in related applications. As described below, data messages passing through a pool are generally described as "proteins".

The example multi-process interactive system 100 includes or spans two computers 101 and 102, but is not limited to two computers, and can run across any number and/or combination of processing systems. In this example, the first computer 101 hosts the process of implementing two squares S (eg S21 , S22 ) and the second computer 102 hosts the process of implementing four squares S (eg S11 , S12 , S13 , S14 ). Alternative embodiments may include any number of square processes S running on any number of computers. The first computer 101 is coupled to a single display 110 and the second computer 102 is coupled to three displays 121 , 122 , 123 . Alternative embodiments may include any number of displays coupled to any number of computers.

Each of the two computers 101 and 102 hosts at least one "mouse" process M (eg M1, M2). The mouse process M includes a high-level driver that transforms computer mouse input events into a suitable stream of user input proteins and passes these proteins to at least one "user input" pool Ui. The gesture/space system packaged as a gesture/space process G (as described in detail below) also passes user input proteins to the user input pool Ui.

Each of the two computers 101 and 102 also hosts at least one "pointer" process P (eg P1, P2). The pointer process P fetches or receives data from the Ui pool and is responsible for determining where the user is directing the pointer to "focus" and for drawing or rendering the appropriate pointer graphics. Pointer process P places data relating to or representing pointer positions and modes into a "coordinating" pool Coo. Pointer process P passes graphics output to a "frame" pool, which is a dedicated abstraction described in detail below.

Furthermore, each of the two computers 101 and 102 hosts several "square" processes S as described above. Each square process S consults the coordinating pool Coo for pointer data and the state of the peer square process S. Each square process S also places data describing their own space and mode state back to the coordinating pool Coo. A square process S passes graphics output to the "frames" pool. Frame pools are specialized abstractions, described in detail below.

The gesture/space process G along with the user input pool Ui and the coordination pool Coo can be hosted on either of the two computers 101 and 102 . Alternatively, the mastership of the pose/space process G together with the user input pool Ui and the coordination pool Coo can be shared between the two computers 101 and 102 . As another alternative configuration, the gesture/space process G along with the user input pool Ui and the coordination pool Coo may be hosted on other computers (not shown).

For proteins deposited into the local frame pool, the system 100 includes a dedicated compositing process com that combines the frame layers into a single frame of output for each display many times per second. It is usually a system-level configuration choice to set the overall display frame rate, but each of the individual processes that make up the square application can use a different frame rate. The compositing process com takes care of matching the frame layers appropriately.

FIG. 2 is a flowchart 200 of the operation of a multi-process interactive system, according to an embodiment. Operations include executing a plurality of processes on at least one processing device 202 . The multiple processes include separable program execution contexts for multiple applications such that each application includes at least one process. Operations include translating events for each of the plurality of processes into data capsules 204 . A data message includes state information for the process that originated the data message and an application-independent representation of the event data for the event. Operations include transmitting 206 the data message to at least one pool of the plurality of pools. Each process operates as a recognition process such that the recognition process recognizes, in a plurality of pools, a data capsule 208 comprising at least one of content corresponding to an interaction function of the recognition process and an identification of the recognition process. The identification process retrieves identified data capsules from a plurality of pools and performs processing 210 appropriate to the contents of the identified data capsules. Operation of the multi-process interactive system enables coordination among processes, wherein the coordination includes each process of the plurality of processes coordinating with a peer process of the plurality of processes by retrieving state information of the peer processes from the plurality of pools. The operation also enables the generation of outputs of the plurality of processes by interactively combining the contents of the collection of data capsules of at least one of the plurality of pools.

In process mouse and gesture/spatial input, the mouse process M monitors the low-level mouse hardware and translates traditional mouse driver events into screen-independent proteins. According to the following description, the mouse process M protein of the embodiment passed to the user input pool Ui is as follows:

descriptions:

-input driver

-mango.local

-mouse

ingests:

-id: 0x012345

- pos: [1280, 800]

-buttons: [1]

The attitude/space process G protein will look similar, as follows:

descriptions:

-input driver

-mango.local

-hand

ingests:

-id: 0x017845

-gripe: ..|||

- pos: [127.4, 10.5, 12.2]

-point: [[-0.099833, 0.000000, -0.995004],

[0.099335 0.995004 -0.009967]]

The pointer process P interprets these messages as hinting at the position of the individual pointers in the three-dimensional space they are responsible for rendering. A static collection of pointers may have been defined in the application code, or an earlier protein may have defined and initialized the pointers.

For the instance of the square program running within the spatial operating environment, each of the pointer processes P knows the exact real-world position of the display screen attached to the computer hosting them. Furthermore, these display screens may have been initialized at startup or dynamically via data messages.

As proteins arrive at user input pool Ui, pointer process P reacts by constructing new proteins and passing them to coordination pool Coo as follows:

descriptions:

-pointer

-click

-mouse

ingests:

-mid: 0x2345

-origin: [0.0, 0.0, 0.0]

-passes-through: [0.0, 1625.6, -2349.3]

descriptions:

-pointer

-click

-hand

-one-finger

ingests:

-mid: 0x2372

-origin: [127.4, 10.5, 12.2]

-passes-through: [0.0, 1625.6, -2349.3]

These proteins or messages define the position of the pointer object with respect to the available displays. Each pointer process P is configured to manage mathematical transformations for displays attached only to the computer on which it is hosted.

Periodically, each pointer process P also draws frames of graphics output. This graphics data is passed to the frame pool. Each frame produced by pointer process P renders all pointer graphics that will appear on the display attached to the computer hosting the process.

With the application model and graphics of an embodiment, the square process S is responsible for tracking and drawing the translucent square that is the focus of the square application. Each square has a position, orientation, size and color. At any time the state of the square changes, the process S of the square puts the protein into the coordinating pool Coo:

descriptions:

-tsquare

-position

ingests:

-tid: 0x45878912

- pos: [0.0, 0.0, 0.0]

-up: [0.0, 1.0, 0.0]

-over: [1.0, 0.0, 0.0]

-size: 25.0

-color: [1.0, 1.0, 1.0, 1.0]

The square process S also passes graphics output to the frame pool, mostly as the pointer process does. However, each square process S renders its own square, whether or not that square will appear on a display attached to the computer hosting the process. Frame processing is described in detail below.

Placing and passing the proteins describing the state of the pointers and squares into the multi-user pool allows the separate processes that make up the application to coordinate, thereby providing coordination between dissimilar processes. The square process S of an embodiment monitors proteins that indicate pointers entering the region of the border of the square. When this happens, the square process S puts the protein into the pool indicating the overlap, and references the squares and pointers involved, as follows:

descriptions:

-tsquare

-pointer-overlap

-entrance

ingests:

-tid: 0x45878912

-mid: 0x2372

Pointer process P monitors this form of protein. When pointer process P recognizes or sees an overlap protein referencing its own mid, it changes the graphical representation it uses when drawing pointer frames. The overlap indicator graph will be used until the process sees the appropriate overlap-exit protein, e.g.:

descriptions:

-tsquare

-pointer-overlap

-exit

ingests:

-tid: 0x45878912

-mid: 0x2372

Many variations of the above coordination strategies are also possible. Pointer process P may handle the responsibility of checking geometric overlap (instead of square process S doing this as described above). Processes can all be frame-synchronized, and overlapping proteins can be generated for each frame, which would eliminate the need for separate entry and exit proteins.

There is almost always diversity in the solutions available to any given coordination problem faced when working within the mechanisms described here. In fact, this flexibility is one of the strengths of this embodiment. The description herein illustrates at least one of the implemented solutions to several interlocking problems faced in attempting to build typical multi-process graphics applications. Several references collect useful messaging patterns, many of which are applicable, eg "Enterprise Integration Patterns: Designing, Building and Deploying Messaging Solutions" by G. Hohpe and B Woolf in ISBN 0321146530.

According to an embodiment, in connecting the user input to the manipulation action, interactively moving the square includes: the square process S uses the data put into the coordination pool Coo by the pointer process P. The square process S initiates movement in response to the identified pointer overlap condition and protein combination with "pointer" and "click" descriptions. While the movement is in progress, the graphical representation of the square changes and the position of the square in space follows the pointer. Movement continues until the corresponding "pointer"/"unclicked" protein is generated.

When the squares of the embodiment overlap each other, they also change their color. Whenever the S process sees a "tsquare"/"position" protein, it calculates whether there is any overlap between itself and the custody of that protein. If present, it uses the overlap indication color when it renders its next frame, otherwise, it uses the normal color.

Note that the flexibility of the loosely coupled architecture of an embodiment provides additional or alternative ways of achieving this behavior. For example, a square process S could avoid doing the overlap calculation and instead offload this work to another process, e.g. that process would continuously do the matching for some or all squares and put the proteins describing the overlap conditions into the coordinating pool Coo. Square process S will simply wait for these proteins:

descriptions:

-tsquare

-square-overlap

ingests:

-tids: [0x45878912, 0x45878916].

This flexibility to slice and dice the application workload is very useful. Computationally intensive tasks can be moved to processors or machines with spare capacity. Producers of data can instantiate helper processes as needed (and terminate them when they are no longer needed). More computing and rendering resources can be applied to areas of the application where the user directly interacts with it, or where the user immediately perceives greater granularity, detail, or refresh rate.

This is all possible because the multi-process interaction system described herein externalizes application state and allows multiple processes to access that state. In contrast, with contemporary programming models, the runtime state is always fully "locked" within a single process.

Multi-process interaction systems encourage programmers to expose all interaction functions as protein-drivable states. Application programming interfaces (APIs) are defined by proteins recognized by each process, rather than traditional function calls. For example, to define a protein that changes the color of any (or all) squares:

descriptions:

-tsquare

-color-change-command

ingests:

-tids: [0x0]

-normal-color: [1.0, 0.0, 0.0, 0.65]

-overlap-color: [1.0, 0.0, 0.0, 1.0]

When any square process S sees the protein in the coordination pool Coo, it checks the tids linked list to determine whether its own unique object id or general address 0x0 exists. If present, the process starts rendering its square using the two new colors specified (one for the normal case and one for the overlapped case).

Using this mechanism and this "expose all interaction functions as proteins" approach, after all other code for the square application has been completed, deployed and running, a new utility for controlling the color of the square can be written. No recompilation or relinking is required to add new functionality to a running application.

Interactive debuggers for graphical applications are another type of program that would benefit from this approach. Before traditional debuggers can reveal much of a program's internal state, they often need to halt the program. However, if all of the program's manipulable state is exposed via the pool described herein, the debugger can monitor and manipulate the program state while the program is running.

Pointer process P and square process S push graphics data to the frame pool so that any display output becomes visible to the user. Embodiments described herein include a variety of ways for outputting graphics, some of which are described in detail herein. Other embodiments may operate under different combinations of the processes described below for pushing graphics data to the frame pool and outputting graphics.

In an embodiment, a process may draw directly to the system graphics layer using a direct rendering framework such as OpenGL. Under this approach, the pool is used for coordination, but not for graphics commands or pixel data.

Another embodiment outputs graphics data via a process that passes rendering commands to the pool. Another process (or processes) is then responsible for interpreting rendering commands and driving the system graphics layer. For example, these commands can be very low-level, such as exposed OpenGL calls. Instead, these rendering commands can be very high-level, such as, for example, the tsquare protein described above, which includes enough information that a dedicated rendering process can draw a square frame by frame.

Yet another embodiment outputs the graphics data via the process of rendering to a pixel buffer in memory, then transfers or places the resulting raw frame data into the pool. Another process (or processes) combines the raw frame data. With this approach, the amount of data processed by the pool is typically much greater than with the graphical output approach described above. However, local rendering and network frame delivery provide a lot of flexibility, so if a high-bandwidth network and fast pooling implementation is available, it is generally used.

The example system 100 described above with reference to FIG. 1C typically outputs graphics data via the process of rendering to a pixel buffer in memory, and then transfers the resulting raw frame data to the pool. Another process (or processes) assembles the raw frame data. With this approach, the amount of data processed by the pool is typically much greater than with the graphical output approach described above. However, local rendering and network frame delivery provide a lot of flexibility, so if a high-bandwidth network and fast pool implementation is available, it is generally used.

Thus, each pointer process P and square process S renders their own separate graphic elements. Each process picks multiple color components and multiple pixels to render. A process can render a full display of (eg 2560x1600) pixels using the RGBA (red, green, blue, alpha) color space or components of the RGB color model with alpha blending and alpha compositing. However, to save computation cycles, rendering overhead, and pool bandwidth, a process can generate only the necessary pixels to capture the projected bounding box of a particular graphics object, and only use two components if the luma (with transparency) rendering is sufficient.

Rendered pixel data is transferred or passed into the frame pool along with various metadata such as geometric extents, layering information, frame rate indication, additional color information, etc. With a square application running in the context of a spatial operating environment, each process has access to real-world geometric data and is able to pass appropriate output to each of the frame pools. This may include rendering more than one frame per output cycle.

Pooling proteins into the local frame pool occurs at a rate that typically does unnecessary compression of pixel data. However, to achieve relatively low latency for interactive applications, network storage can reduce the amount of data sent for each frame. Embodiments use hardware compression to reduce the number of bytes required to represent each array of pixels, but embodiments are not limited thereto.

Referring to Figure 1C, an embodiment of the system 100 uses a dedicated compositing process COM that combines these frame layers many times per second into a single frame for each display's output. It is usually a system-level configuration choice to set the global display frame rate, but each of the individual processes that make up the square application can use a different frame rate. The compositing process com takes care of matching frame layers appropriately.

As described above with reference to FIGS. 1A-1C , the multi-process interaction system of the embodiment includes processes, pools and proteins. Solid rectangles in the system represent processes, while ovals represent pools, mechanisms for structured data exchange. Data messages passing through a pool are often described as "proteins". Each of the processes generates proteins and deposits proteins into and retrieves proteins from one or more pools.

Pools and proteins are components of the methods and systems described herein for encapsulating data to be shared between or across processes. These mechanisms include slaws (multiple "slaws") in addition to proteins and pools. In general, polysaras provide the lowest-level data definitions for interprocess exchange, proteins provide intermediate-level structures and hooks for querying and filtering, and pools provide high-level organization and access semantics. Multisala includes mechanisms for efficient, platform-independent data representation and access. Proteins provide data encapsulation and delivery schemes using slaws as payloads. Pooling provides structured and flexible aggregation, sorting, filtering, and distribution of proteins within-process, between local processes, across a network between remote or distributed processes, and via long-term (eg, on-disk, etc.) storage.

The configuration and implementation of an embodiment of the multi-process interaction system includes several constructs that together enable multiple capabilities. For example, embodiments described herein provide efficient exchange of data between a large number of processes as described above. Embodiments described herein also provide flexible data "types" and structures so that a wide variety of data types and uses are supported. Furthermore, embodiments described herein include flexible mechanisms for data exchange (eg, local storage, disk, network, etc.), all driven through substantially similar application programming interfaces (APIs). Furthermore, the described embodiments enable data exchange between processes written in different programming languages. Furthermore, embodiments described herein enable automatic maintenance of data caching and convergence state.

3 is a block diagram of a processing environment including data representation using slaws, proteins, and pools, according to an embodiment. The main configurations of the embodiments presented here include multi-salads (multiple "salads"), proteins, and pools. The polysala described here includes mechanisms for efficient, platform-independent data representation and access. The proteins described in detail herein provide data encapsulation and transmission schemes, and the payload of the example proteins includes slaw. The pools described here provide structured yet flexible aggregation, ordering, filtering and distribution of proteins. Pools provide access to data by means of proteins within a process, between local processes, across a network between remote or distributed processes, and via "long-term" (eg, on-disk) storage.

Figure 4 is a block diagram of a protein according to an embodiment. As described in detail below, a protein includes a length header, description, and uptake. Each of the description and intake includes a salad or multi-salad.

Figure 5 is a block diagram of a description according to an embodiment. As described in detail below, descriptions include offset, length, and slaw.

Figure 6 is a block diagram of ingestion, under an embodiment. As detailed below, intake includes offset, length, and slaw.

7 is a block diagram of a salad, under an embodiment. As described in detail below, a slaw includes a type header and type-specific data.

Figure 8A is a block diagram of proteins in a pool, according to an embodiment. A protein includes a length header ("Protein Length"), Descriptor Offset, Ingest Offset, Descriptor and Ingest. Descriptions include offset, length, and slaw. Intake includes offset, length and slaw.

The proteins described here are mechanisms for encapsulating data that needs to be shared between processes or moved across a bus or network or other processing structure. As an example, proteins provide an improved mechanism for transmitting and manipulating data, including data corresponding to or associated with user interface events, in particular the user interface events of the embodiments included in U.S. Patent No. 7,598,942 User Interface Events for the Gesture Interface described and incorporated herein by reference in its entirety. As another example, proteins provide improved mechanisms for transmitting and manipulating data including, but not limited to, graphical data or events and state information, just to name a few. A protein is an associated collection of structured record formats and methods for manipulating records. As used herein, manipulation of records includes putting data into structures, getting data out of structures, and querying the format and existence of data. Proteins are configured for use via code written in a variety of computer languages. As described herein, proteins are also configured as basic building blocks for pools. Furthermore, proteins are configured to be naturally able to move between processors and across networks while leaving the data they comprise intact.

In contrast to traditional data transfer mechanisms, proteins are untyped. Although untyped, proteins provide powerful and flexible pattern-matching tools on which to implement "type-alike" functions. Proteins configured as described herein are also inherently multipoint (although point-to-point forms are readily achievable as a subset of multipoint transmission). Furthermore, for example, proteins define a "universal" recording format that does not differ (or differs only in the type of optional optimizations performed) between eg in-memory, on-disk and online (network) formats.

Referring to Figures 4 and 8, the proteins of the embodiments are linear sequences of bytes. Within these bytes encapsulate a linked list of descriptions and a collection of key-value pairs called ingestions. The description list includes arbitrarily fine-grained yet efficiently filterable per-protein event descriptions. Ingestion consists of a collection of key-value pairs containing the actual content of the protein.

The association of proteins with key-value pairs and some core ideas about network-friendliness and multipoint data exchange is shared with earlier systems (eg Linda, Jini) that privileged the concept of "tuples". Proteins differ from tuple-oriented systems in several major ways, including the use of lists of descriptions to provide a standard, optimizeable pattern-matching substrate. Proteins also differ from tuple-oriented systems in a strict specification of a record format suitable for a variety of storage and language constructs, along with a few specific implementations of an "interface" to that record format.

Turning to the description of the protein, the first four or octets of the protein specify the length of the protein, which in an embodiment must be a multiple of 16 bytes. The 16-byte granularity ensures byte alignment and bus alignment efficiency is achievable on contemporary hardware. Proteins that are not naturally "quad-aligned" are padded with arbitrary bytes such that their length is a multiple of 16 bytes.

The length part of a protein has the following format: in big-endian format, 32 bits specifying the length, where the four lowest-order bits serve as flags to indicate macro-level protein structural properties; if the length of the protein is greater than 2^ 32 bytes, followed by 32 additional bits.

The 16 byte alignment restriction of an embodiment means that the lowest order bits of the first four bytes can be used as flags. And thus, the first three low-order bit flags indicate, respectively, whether the length of the protein can be represented in the first four bytes or requires eight, whether the protein uses little-endian or little-endian byte ordering, and whether the protein is Whether a standard or non-standard structure is adopted, but the protein is not limited to this. The fourth flag bit is reserved for future use.

If the eight-byte length flag is set, by reading the next four bytes and using them as the high-order byte of an eight-byte integer with big-endian priority (where the four bytes already read provide the low-order part) to calculate the length of the protein. If the little-endian flag is set, all binary numeric data in the protein will be interpreted as little-endian (otherwise big-endian). If the non-standard flag bit is set, the rest of the protein does not conform to the standard structure described below.

In addition to saying that there are various methods for using proteins and pools to describe and synchronize non-standard protein formats available to system programmers, and that these methods can be useful when space or computer cycles are constrained, here Non-standard protein structures are not discussed further. For example, the shortest protein of an embodiment is sixteen bytes. A standard format protein cannot fit any actual payload data into these sixteen bytes (the lion's share of which has been shifted to describe the location of the protein's constituent parts). But a protein in a non-standard format could conceivably use 12 of its 16 bytes for data. Two applications exchanging proteins can mutually decide that any 16-byte long protein they send always includes 12 bytes representing, for example, 12 8-bit sensor values from a real-time analog-to-digital converter.

In the standard structure of proteins, following the length header, two other variable-length integers appear. These numbers specify the offset to the first element and first key-value pair (ingest) in the description list, respectively. These offsets are also referred to herein as descriptive offsets and ingest offsets, respectively. The byte order of each quadrangle of these numbers is specified by the protein endianness flag. For each, the most significant bit of the first four bytes determines whether the number is four or eight bytes wide. If the most significant bit (msb) is set, the first four bytes are the most significant bytes of the doubleword (eight byte) number. This is referred to herein as an "offset form". The use of separate offsets pointing to descriptions and pairs allows descriptions and pairs to be processed by different code paths, enabling certain optimizations related to, for example, description pattern matching and protein assembly. The presence of these two offsets at the beginning of the protein also allows several useful optimizations.

Most proteins will not be so large as to require eight bytes of length or pointers, so usually the length (with flags) and two offset numbers will only occupy the first three bytes of the protein. In many hardware or system architectures, the fetching or reading of a certain number of bytes beyond the first is "free" (for example, 16 bytes take exactly the same number of clock cycles as a single byte to process across units device's main bus pull (pull)).

In many instances, it is useful to allow implementation-specific or context-specific caches or metadata inside proteins. The use of offsets allows to create arbitrarily sized "holes" near the beginning of the protein, into which such metadata can be inserted. Implementations that can take advantage of eight bytes of metadata get those bytes for free on many system architectures, where the protein's length header is taken each time.

The description offset specifies the number of bytes between the start of the protein and the first description entry. Each descriptor includes an offset to the next descriptor (in offset format, of course), followed by a variable-width length field (also in offset format), followed by a slaw. If nothing else is specified, the offset is, by rule, four bytes of zero. Otherwise, the offset specifies the number of bytes between the start of this descriptor and the next descriptor. The length field specifies the length of the salad in bytes.

In most proteins, each description is a string formatted in a salad string fashion: a four-byte length/type header with the most significant bit set and only the lower 30 bits used to specify the length, followed by the header section indicates the number of data bytes. Normally, the length header takes its byte order from the protein. The bytes are assumed to encode UTF-8 characters (so - note (nota bene) - the number of characters is not necessarily the same as the number of bytes).

The ingestion offset specifies the number of bytes between the start of the protein and the first ingestion entry. Each ingest item includes an offset (in offset format) to the next ingest item, again followed by a length field and a slaw. An ingest offset is functionally identical to a describe offset, except that it points to the next ingest item instead of the next describe item.

In most proteins, each ingest has a salad cons type that includes a linked list of two values usually used as key/value pairs. A shara cons record includes a four-byte length/type header with the second most significant bit set and only the lower 30 bits used to specify the length; a four-byte offset to the start of the value (second) element; A key element of four bytes in length; a slaw record for a key element; a value element of four bytes in length, and finally a slaw record for a value element.

Usually, cons keys are salad skewers. Replication of data across several protein and salad cons length and offset fields also provides more opportunities for refinement and optimization.

As mentioned above, constructs for embedding typed data within proteins according to embodiments are tagged byte sequence specifications and abstractions called "salads" (plural is "salads") . A slaw is a linear sequence of bytes representing a piece of (possibly aggregated) typed data, and is associated with a programming language-specific API that allows the creation, modification, and movement of multiple salad. The Salad type scheme is intended to be extensible and as lightweight as possible, and to be a common base usable from any programming language.

The desire to build an efficient and large-scale inter-process communication mechanism is the driving force of the slaw configuration. Traditional programming languages provide complex data structures and type facilities that work well in process-specific memory layouts, but when data is moved between processes or stored on disk, these data representations are always corrupted. The shara schema is the premier essentially efficient multi-platform friendly low-level data model for inter-process communication.

But more importantly, the slaw is configured to influence and enable the development of future computing hardware (microprocessors, memory controllers, disk controllers) along with proteins. A few specific additions to the instruction set of eg publicly available microprocessors make it possible for multi-sharara to become even for a single process, an in-memory data layout as efficient as the scheme used in most programming languages.

Each slaw consists of a variable-length type header, followed by a type-specific data layout. In example embodiments that support full slaw functionality in, for example, C, C++, and Ruby, the type is indicated by a generic integer defined in a system header file accessible from each language. More complex and flexible type resolution functions can also be implemented: for example, indirect types via generic object IDs and network lookups.

For example, the slaw configuration of an embodiment allows slaw records to be used as objects from both Ruby and C++ in a language-friendly manner. A suite of utilities external to the C++ compiler fully inspects the slaw byte layout, creates header files and macros specific to the individual slaw types, and automatically generates bindings for Ruby. As a result, well-configured salad types are very efficient even when used from within a single process. Any slaw anywhere in a process's accessible memory can be addressed without copying or "deserialization" steps.

The salad function of an embodiment includes an API facility for performing one or more of: creating a new salad of a specialized type; creating or constructing a language-specific reference to a salad from bytes on disk or in memory; methods to embed data in the salad, query the size of the salad; retrieve data from the salad; clone the salad, and translate the byte order and other format properties of all data in the salad. Each salad performs the above behavior.

Figure 8B illustrates a salad header format, according to an embodiment. A detailed description of the salad follows.

The internal structure of each salad optimizes each of type resolution, access to encapsulated data, and size information for that salad instance. In an embodiment, the full set of salad types is done minimally by design, and includes: salad strings, salad cons (ie, dyads); salad lists; Permutations or broad collections of individual numeric types of this basic property. The other essential quality of any salad is its size. In an embodiment, polysaras have byte lengths quantized to multiples of four; these four-byte words are referred to herein as "quads." In general, such quad-based sizes align polysara well with the configuration of modern computer hardware architectures.

The first four bytes of each slaw in an embodiment include a header structure that encodes a type description and other meta-information, and attributes a specific type meaning to a particular bit pattern. For example, the first (most significant) bit of the slaw header is used to specify whether the size of the slaw (the length in a quad word) follows the original four-byte type header. When this bit is set, it is understood that the size of the slaw is explicitly recorded in the next four bytes of the slaw (eg, bytes five through eight); if the size of the slaw cannot be represented by four bytes ( That is, if the size is equal to or greater than two to the thirty-second power), the next most significant bit of the initial four bytes of the slaw is also set, which indicates that the slaw is eight bytes (rather than four bytes) long. In this case, the checking process will find the length of the slaw stored in sequential bytes five through twelve. On the other hand, a small number of slaw types means that in many cases the fully specified typical bit pattern "leaves unused" many bits in the four-byte slaw header, and in such cases, these bits can be used for Encodes the length of the slaw, saving bytes (five to eight) that would otherwise be required.

For example, an embodiment leaves the most significant bit of the slaw header (the "length obeys" flag) unset, and sets the next bit to indicate that the slaw is a "wee cons", in which case the remaining three The length of the slaw encoded (in 4-tuples) in tens. Similarly, "wee string" is marked by pattern 001 in the header, which leaves twenty-nine bits for the length of the salad string; bootstrap 0001 in the header describes "wee list ", which can be represented by twenty-eight available length bits, which can be a slaw list of sizes up to two to twenty-eight quadruples. A "full string" (or cons or linked list) has a different bit signature in the header, where, since the slaw length is encoded separately in bytes five to eight (or twelve in extreme cases), So the most significant header bit must be set. Note that the plasma implementation "decides" at salad build time whether to take "tiny" or "full" versions of these constructs (the decision is based on whether the resulting size will "fit" the available tiny bits), but full contrast to Minute details are hidden from the user of the plasma implementation, the user only knows and cares only that she is using salad skewers, or salad cons, or salad lists.

In an embodiment, the value slaw is indicated by the leading header pattern 00001. Subsequent header bits are used to represent sets of orthogonal features that can be combined in arbitrary permutations. Embodiments employ, but are not limited to, five such character bits to indicate whether a number is: (1) floating point; (2) complex; (3) unsigned; (4) "wide"; "stumpy" (permutations of (4) "wide" and (5) "stumpy" are represented as numbers indicating eight, sixteen, thirty-two, and sixty-four digits). Two additional bits (e.g. (7) and (8)) indicate that the packed numeric data is a two-, three-, or four-element vector (where two bits of zero indicate that the value is a "one-element vector" (i.e., a scalar)) . In this embodiment, eight bits of the fourth header byte are used to encode the size (in bytes, not quads) of the packed numeric data. The encoding of this size is offset by one so that it can represent any size between one and two hundred and fifty-six bytes and in between. Finally, two character bits (eg (9) and (10)) are used to indicate that the numeric data encodes an array of individual numeric items, each of which is of the type described by the character bits (1) to (8). In the case of arrays, the individual value items are not each tagged with another header, but are packed as contiguous data following a single header and possibly explicit slaw size information.

This embodiment provides simple and efficient salad copying (which can be implemented as a byte-for-byte copy) and extremely direct and efficient salad comparison (in this embodiment, if and only if there are A one-to-one match of each of the constituent bytes, the two polysaras are the same). For example, the latter property is important for the efficient implementation of protein architectures, one of the key and pervasive features of protein architectures being the ability to search or "match" descriptive linked lists of proteins.

Furthermore, embodiments herein allow simple and efficient construction of aggregate salad forms (eg, salad cons and salad chain lists). For example, an embodiment constructs salad cons (which can be of any type, including self-aggregate) from two constituent multisala by the following steps: (a) query the size of each constituent salad, (b) allocate a size equal to the two constituent multisala The size of the header plus the memory for the sum of one, two, or three quadruples required for the size structure. (c) record the slaw header (plus size information) in the first four, eight or twelve bytes; and then (d), copy the bytes that make up the slaw to the next memory . Importantly, such a constructor need not know anything about the two types that make up the polysara, only their size (and accessibility as a sequence of bytes) matters. The same process applies to the construction of slaw-lists, which are sorted wrappers of (possibly) many sub-multi-salads of (possibly) heterogeneous types.

Another consequence of the basic format of the slaw system as sequential bytes in memory is obtained in conjunction with the "traversal" activity - the cyclic usage pattern uses, for example, sequential access to individual multi-salads stored in a slaw-linked table. Individual slaws representing descrip- tions and uptake within protein structures must similarly be traversed. Such a transfer is done in an extremely straightforward and efficient manner to "reach" the next salad in the salad chain list, adding the length of the current salad to its location in memory, and the resulting memory location is likewise the head of the next salad . Such simplifications are possible because Salad and Protein Design eschew "indirection"; there are no pointers; instead, the data simply exists in its entirety in place.

On the point of salad comparison, a full implementation of a plasma system must acknowledge the existence of different and incompatible data representation schemes across and among different operating systems, CPUs, and hardware architectures. Most of these differences include byte ordering strategies (such as little-endian vs. big-endian) and floating-point representation, but there are others as well. The Plasma specification requires ensuring that data packaged in a multi-sala is interpretable (i.e. must appear in the native format of the schema or platform from which the slaw is checked). This requirement in turn means that the plasma system itself is responsible for the data format conversion. However, the specification only states that the transition occurs only before the slaw becomes "at all visible" to an executing process that can inspect it. So it's up to the individual implementation, at which point, it chooses to perform this format-c conversion; the two appropriate methods are the data format of the slaw data payload conforming to the local schema (1) since the slaw alone is where it's already packed "Pull out" of the protein, or (2) all salads used in the protein at the same time, due to the extraction of the protein from the pool in which it resides. Note that conversion constraints allow for the possibility of hardware-assisted implementations. For example, a networking chipset built with display plasma capabilities may choose to perform format conversion intelligently and "on the fly" based on known properties of the receiving system. Alternatively, the transmitting process can convert the data payload into the canonical format, while the receiving process symmetrically converts from the canonical format to the "native" format. Another embodiment performs format conversion "at the metal", meaning that data is always stored in a canonical format, even in local memory, and when data is retrieved from memory and placed in the register closest to the CPU In , the memory controller hardware performs the translation itself.

A minimal (and read-only) protein implementation of an embodiment includes operations or behaviors in one or more applications or programming languages that utilize the protein. Figure 8C is a flowchart 850 for using proteins, under an embodiment. The operation begins by querying the length in bytes 852 of the protein. The number 854 of query description items. The number 856 of queries ingested. The description item 858 is searched by the index number. Ingestion 860 is retrieved by index number.

Embodiments described here also define basic methods that allow proteins to be constructed and populated with data, helper methods for programmers to more easily perform common tasks, and hooks for creating optimizations. Figure 8D is a flowchart 870 for constructing or generating proteins, according to an embodiment. Operation begins with creating 872 a new protein. A series of description items 874 are added. Ingest 876 additionally. The query matches the existence 878 of the description, and the query matches the existence 880 of the ingestion key. Given an ingestion key, an ingestion value is retrieved 882 . Pattern matching is performed 884 across descriptions. Embed unstructured metadata 886 near the beginning of the protein.

As mentioned above, polysaras provide the lowest-level data definitions for inter-process exchange, proteins provide mid-level structures and hooks for querying and filtering, and pools provide high-level organization and access semantics. Pools are storage for proteins, providing linear serialization and state caching. Pools also provide multi-process access by multiple programs or applications of a large number of different types. Additionally, pools provide a common set of optimizeable filtering and pattern matching behaviors.

The pool of an embodiment, which can hold tens of thousands of proteins, is used as holding state so that individual processes can offload much of the lengthy bookkeeping common to multi-process program code. The pool holds or holds a large buffer of past proteins available—Platonic pools are explicitly infinite—so that participating processes can scan backwards and forwards in the pool at will. Of course, the size of the buffer is implementation dependent, but in typical use it is generally possible to keep proteins in the pool for hours or days.

In contrast to the mechanical peer-to-peer approach taken by existing inter-process communication frameworks, the most common style used by the pools described here adheres to biological metaphors. The name protein alludes to biological implications: data proteins in pools can be used for flexible querying and pattern matching by massive computational processes, just as chemical proteins in living tissue can be used for pattern matching and filtering by huge numbers of cellular reagents.

Two additional abstractions rely on biological metaphors, including the use of "handlers" and the Golgi framework. Processes participating in a pool typically create multiple handles. Handles are relatively small bundles of code that associate matching conditions with processing actions. By associating one or more handles to the pool, the process sets flexible callback triggers that encapsulate state and react to new proteins.

Processes participating in several pools usually inherit from the abstract Golgi class. The Golgi framework provides several useful routines for managing multiple pools and handles. The Golgi class also encapsulates parent-child relationships, providing a mechanism for local protein exchange without using pools.

The pooling API provided according to the embodiments is configured to allow pooling to be implemented in a variety of ways to take into account both system-specific goals and the capabilities available for a given hardware and network architecture. The two fundamental system provisions upon which pools depend are the storage facility and the means for interprocess communication. Existing systems described herein use a flexible combination of shared memory, virtual memory, and disk for storage tools, and IPC queues and TCP/IP sockets for inter-process communication.

Pool functions of an embodiment include, but are not limited to, the following: join a pool; place proteins in a pool; retrieve the next unseen protein from a pool; rewind or fast-forward content (eg, proteins) within a pool. Additionally, pooling functions may include, but are not limited to the following: setting up stream pool callbacks for processes; selectively retrieving proteins matching a particular pattern of descriptive or ingestion keys, scanning backwards and forwards for proteins matching a particular pattern of descriptive or ingestion keys protein.

The proteins described above are provided to the pool in such a way that the protein data content is shared with other applications. 9 is a block diagram of a processing environment including data exchange using slaws, proteins, and pools, under an embodiment. This example environment includes three devices (eg, device X, device Y, and device Z, collectively referred to herein as "devices") that share data using the slaw, protein, and pool described above. Each of the devices is coupled to three pools (eg pool 1, pool 2, pool 3). Pool 1 includes multiple proteins (e.g., protein X1, protein Z2, protein Y2, protein X4, protein Y4) contributed or transferred to the pool from various devices (e.g., device Z transferred or contributed protein Z2 to pool 1, etc.). Pool 2 includes a plurality of proteins (eg, protein Z4, protein Y3, protein Z1, protein X3) contributed or transferred to the pool from various devices (eg, device Y transferred or contributed protein Y3 to pool 2, etc.). Pool 3 includes a plurality of proteins (Protein Y1, Protein Z3, Protein X2) contributed or transferred to the pool from various devices (eg, device X transfers or contributes protein X2 to Pool 3, etc.). While the example described above includes three devices coupled or connected in three pools, any number of devices may be coupled or connected in any number of pools in any manner or combination, and any pool may include devices from any number or Any number of proteins contributed by the combined device. The proteins and pools of this example are as described above with reference to Figures 3-8.

10 is a block diagram of a processing environment including multiple devices and a number of programs running on one or more of the devices where plasma constructs (e.g., pools, proteins, and salads) are used, according to an embodiment Designed to allow a large number of running programs to share and collectively respond to events generated by a device. This system is only one example of a multi-user, multi-device, multi-computer interactive control scenario or configuration. More specifically, in this example, an interactive system that includes multiple devices (eg, devices A, B, etc.) and multiple programs (eg, applications AA-AX, applications BA-BX, etc.) running on the devices uses Plasma constructs (such as pools, proteins, and salads) to allow running programs to share and collectively respond to events generated by these input devices.

In this example, each device (eg, device A, B, etc.) translates discrete raw data generated by or output from programs running on the respective device (eg, applications AA-AX, applications BA-BX, etc.) into plasma proteins, and store these proteins in the plasma pool. For example, program AX generates data or output and provides that output to device A, which in turn translates the raw data into proteins (eg, protein 1A, protein 2A, etc.) and deposits these proteins into a pool. As another example, program BC generates data and provides that data to device B, which in turn translates the data into proteins (eg, protein 1B, protein 2B, etc.) and deposits these proteins into a pool.

Each protein includes a linked list of descriptions specifying the data or output registered by the application as well as identification information for the program itself. Where possible, protein descriptions can also give generic semantic meaning to output events or actions. A protein's data payload (eg, ingestion) carries the entire set of useful state information for program events.

Regardless of the type of procedure or device, the aforementioned proteins are available in pools used by any program or device coupled or connected to the pool. Any number of programs running on any number of computers can extract event proteins from the input pool. These devices need only be able to participate in a pool via a local memory bus or network connection in order to extract proteins from the pool. A direct consequence of this situation is the advantageous possibility of decoupling the processes responsible for generating processing events from the processes consuming or interpreting them. Another consequence is the multiplexing of consumers and sources of events, such that a device can be controlled by one person, or used simultaneously by several people (e.g. plasma-based input framework supports many concurrent users), while the resulting event stream is useful for multiple Event consumers are visible in turn.

As an example, device C may extract one or more proteins (eg, protein 1A, protein 2A, etc.) from the pool. After protein extraction, in a processing event corresponding to the protein data, device C may use the data of the protein retrieved or read from the protein's ingestion and the described slaw. As another example, Device B may extract one or more proteins (eg, Protein 1C, Protein 2A, etc.) from the pool. After protein extraction, device B can use the protein data in processing events corresponding to the protein data.

A device and/or program coupled or connected to the pool can browse backwards and forwards in the pool for specific sequences of proteins. For example, it is often useful to set up a program to wait for a protein matching a particular pattern to appear, and then browse back to see if that protein has already appeared in conjunction with particular other proteins. Such tools for utilizing stored event histories in input pools generally make writing state management code unnecessary, or at least significantly reduce reliance on such undesirable coding patterns.

11 is a block diagram of a processing environment including a plurality of devices and a number of programs running on one or more of the devices in which plasma constructs (e.g., pools, proteins, and salads) according to an alternative embodiment Used to allow a large number of running programs to share and collectively respond to events generated by a device. This system is only one example of a multi-user, multi-device, multi-computer interactive control scenario or configuration. More specifically, in this example, multiple programs running on multiple devices (eg, devices X and Y coupled to devices A and B, respectively) and one or more computers (eg, device A, device B, etc.) Interactive systems (such as Apps AA-AX, Apps BA-BX, etc.) are built using Plasma (eg, Pools, Proteins, and Salads) to allow running programs to share and collectively respond to events generated by these input devices.

In this example, each device (e.g., devices X and Y coupled to devices A and B, respectively) is managed and/or coupled to one or more Runs under or in association with a program that translates discrete raw data generated by the hardware of a device (e.g., device X, device A, device Y, device B, etc.) into plasma proteins and deposits these proteins into a plasma pool middle. For example, device X running in association with application AB hosted on device A generates raw data, translates discrete raw data into proteins (eg, protein 1A, protein 2A, etc.), and deposits these proteins into a pool. As another example, device X running in association with application AT hosted on device A generates raw data, translates discrete raw data into proteins (e.g., protein 1A, protein 2A, etc.), and deposits these proteins into a pool . As yet another example, device Z running in association with an application CD hosted on device C generates raw data, translates discrete raw data into proteins (e.g., protein 1C, protein 2C, etc.), and deposits these proteins into a pool .

Each protein includes a linked list of descriptions specifying the actions registered by the input device as well as identification information for the device itself. Where possible, protein descriptions can also give generic semantic meaning to device actions. The protein's data payload (eg, ingestion) carries the entire set of useful state information for device events.

Regardless of the type of procedure or device, the aforementioned proteins are available in pools used by any program or device coupled or connected to the pool. Any number of programs running on any number of computers can extract event proteins from the input pool. These devices need only be able to participate in a pool via a local memory bus or network connection in order to extract proteins from the pool. A direct consequence of this situation is the advantageous possibility of decoupling the processes responsible for generating processing events from the processes consuming or interpreting them. Another consequence is the multiplexing of consumers and sources of events, such that the input device can be controlled by one person, or used simultaneously by several people (e.g. a plasma-based input framework supports many concurrent users), while the resulting event stream is useful for Multiple event consumers are visible sequentially.

A device and/or program coupled or connected to the pool can browse backwards and forwards in the pool to find specific sequences of proteins. For example, it is often useful to set up a program to wait for a protein matching a particular pattern to appear, and then browse back to see if that protein has already appeared in conjunction with particular other proteins. Such tools for utilizing stored event histories in input pools generally make writing state management code unnecessary, or at least significantly reduce reliance on such undesirable coding patterns.

12 is a block diagram of a processing environment including multiple input devices coupled among a multitude of programs running on one or more devices in which plasma constructs (e.g., pools, protein and Salad) are used to allow a large number of running programs to share and collectively respond to events generated by input devices. This system is only one example of a multi-user, multi-device, multi-computer interactive control scenario or configuration. More specifically, in this example, multiple input devices (eg, input devices A, B, BA, and BB, etc.) and multiple programs running on one or more computers (eg, device A, device B, etc.) An interactive system (not shown) uses plasma builds (eg, pools, proteins, and salads) to allow running programs to share and collectively respond to events generated by these input devices.

In this example, each input device (eg, input devices A, B, BA, and BB, etc.) is managed by a software driver hosted on the corresponding device (eg, device A, device B, etc.) that will be controlled by the input device hardware The generated discrete raw data is translated into plasma proteins and these proteins are deposited into the plasma pool. For example, input device A generates raw data and provides that raw data to device A, which in turn translates discrete raw data into proteins (eg, protein 1A, protein 2A, etc.) and deposits these proteins into a pool. As another example, input device BB generates raw data and provides that raw data to device B, which in turn translates discrete raw data into proteins (e.g., protein 1B, protein 3B, etc.) and deposits these proteins into a pool .

Each protein includes a list of descriptions that specify the actions registered by the input device and the

self-identifying information. Where possible, protein descriptions can also give generic semantic meaning to device actions. The protein's data payload (eg, ingestion) carries the entire set of useful state information for device events.

For illustration, here are example proteins for two typical events in such a system. Proteins are represented here as text, however, in actual implementation, the components of these proteins are typed data bundles (eg salad). The proteins describing the "one finger click" pose (described in a related application) in general are as follows:

As another example, a protein describing a mouse click is as follows:

One or both of the aforementioned sample proteins may cause a participating program of the host device to run a specific portion of its code. These programs may be interested in semantic annotation in general: "point" most generally of all, or more specifically "engage, one". Alternatively, they could look for the event "one-finger-engage", or even a single aggregated object "hand-id-23", which only appears to be generated by the precise device.

Regardless of the type of procedure or device, the aforementioned proteins are available in pools used by any program or device coupled or connected to the pool. Any number of programs running on any number of computers can extract event proteins from the input pool. These devices need only be able to participate in a pool via a local memory bus or network connection in order to extract proteins from the pool. A direct consequence of this situation is the advantageous possibility of decoupling the processes responsible for generating "input events" from the processes that consume or interpret them. Another consequence is the multiplexing of consumers and sources of events, such that the input device can be controlled by one person, or used simultaneously by several people (e.g. a plasma-based input framework supports many concurrent users), while the resulting event stream is useful for Multiple event consumers are visible sequentially.

As an example or protein use, device C may extract one or more proteins (eg, protein 1B, etc.) from the pool. After protein extraction, in processing the input events of the input devices CA and CC to which the protein data corresponds, device C may use the protein's data retrieved or read from the protein's uptake and described slaw. As another example, device A may extract one or more proteins (eg, protein 1B, etc.) from the pool. After protein extraction, device A can use the protein data in processing input events of input device A corresponding to the protein data.

A device and/or program coupled or connected to the pool can browse backwards and forwards in the pool, looking for specific sequences of proteins. For example, it is often useful to set up a program to wait for a protein matching a particular pattern to appear, and then browse back to see if that protein has already appeared in conjunction with particular other proteins. Such tools for utilizing stored event histories in input pools generally make writing state management code unnecessary, or at least significantly reduce reliance on such undesirable coding patterns.

Examples of input devices used in embodiments of the system described herein include gesture input sensors such as those used in consumer electronics, keyboards, mice, infrared remote controls, and task-oriented tangible media objects, just to name a few.

13 is a block diagram of a processing environment including multiple devices coupled among a multitude of programs running on one or more devices in which plasma constructs (e.g., pools, proteins, and salads) are processed according to another alternative embodiment. ) are used to allow a large number of running programs to share and collectively respond to device-generated graphics events. This system is just one example of a system that includes multiple running programs (eg, Figures A-E) and one or more display devices (not shown), where plasma constructs (eg, pool, protein, and salad) are used to make Graphics output from some or all of the programs is available to other programs to allow running programs to share and collectively respond to device-generated graphics events.

It is often useful for a computer program to display graphics generated by another program. A few common examples include video conferencing applications, web-based slideshow and presentation programs, and window managers. In this configuration, the pool serves as a plasma library to implement a common framework that encapsulates video, network application sharing, and window management, and allows programmers to add several features that are not normally available in current versions of such programs.

Programs running in the plasma synthesis environment (eg, Figures A-E) participate in the coordination pool by coupling and/or connecting to the pool. Each program can deposit proteins in this pool to indicate the availability of various graph sources. Programs available for display graphics also deposit proteins to indicate their display capabilities, security and user profiles, and physical and network locations.

Graphics data can also be sent through the pool, or the display program can point to other kinds of network resources (such as RSTP streams). As used herein, the phrase "graphics data" refers to a variety of different representations that rely on a broad continuum; examples of graphics data include, but are not limited to, textual instances (such as "image" or blocks of pixels), procedural instances (such as "drawing " indicates sequences, such as under typical openGL pipeline flow), and descriptive examples (eg, instructions to combine other graphics constructs by means of geometric transformation, clipping, and compositing operations).

On the local machine, graphics data can be passed through platform-specific display driver optimizations. Even when graphics are not being sent via the pool, periodic screen captures are usually stored in the coordination pool so that clients can still display back graphics without directly accessing more esoteric sources.

Unlike most messaging frameworks and network protocols, the multi-process interactive system described herein includes a pool that holds a large buffer of data. Thus, programs can rewind back into the pool to see access and usage patterns (in the case of coordination pools) or to fetch previous graphics frames (in the case of graphics pools).

14 is a block diagram of a processing environment including multiple devices coupled among a multitude of programs running on one or more devices in which plasma constructs (e.g., pools, proteins, and salads) are processed according to yet another alternative embodiment. ) are used to allow status inspection, visualization and debugging of running programs. This system is just one example of a system that includes multiple running programs (eg, program P-A, program P-B, etc.) on multiple devices (eg, device A, device B, etc.), where some programs use or access Access the internal state of other programs.

Most interactive computer systems include many programs running side by side with each other on a single machine or on multiple machines and interacting across a network. Because runtime data is hidden inside each process and difficult to access, multiprogrammed systems can be difficult to configure, analyze, and debug. The general framework and plasma architecture of the embodiments described here allows running programs to have most of their data available via the pool so other programs can check their status. The framework enables debugging tools that are more flexible than traditional debuggers, sophisticated system maintenance tools, and visualization tools configured to allow a human operator to analyze in detail a sequence of states that one or more have passed through.

Referring to FIG. 14, the programs running in the framework (such as program P-A, program P-B, etc.) generate or create a process pool when the program starts. The pool is registered in the system calendar, and security and access controls are applied. More specifically, each device (e.g., device A, B, etc.) translates discrete raw data generated by or output from a program (e.g., program P-A, program P-B, etc.) running on the respective device into plasma proteins, and These proteins are deposited into the plasma pool. For example, program P-A generates data or output and provides that output to device A, which in turn translates the raw data into proteins (e.g., protein 1A, protein 2A, protein 3A, etc.) and deposits these proteins into a pool. As another example, program P-B generates data and provides that data to device B, which in turn translates the data into proteins (eg, proteins 1B-4B, etc.) and deposits these proteins into a pool.

For the duration of the program's life, other programs with sufficient access permissions can attach to the pool and read the program's deposited proteins; this represents a basic checking pattern, and is conceptually a "one-way" or "read-only" proposition: yes An interested entity of program P-A examines the stream of state information that P-A keeps in its process pool. For example, an inspection program or application running under device C may extract one or more proteins (eg, protein 1A, protein 2A, etc.) from the pool. After protein extraction, device C can use the protein's data retrieved or read from the protein's ingestion and described slaw to access, interpret and examine the internal state of program P-A.

However, recalling that the Plasma system is not only an efficient state transfer scheme but also an omni-directional messaging environment, several additional patterns support program-to-program status checking. Authorized checkers can themselves deposit proteins into program P's process pool to influence or control the properties of the state information produced and placed in the process pool (after all, program P not only writes but also reads).

15 is a block diagram of a processing environment including multiple devices coupled among a multitude of programs running on one or more devices in which plasma constructs (e.g., pools, proteins, and salads) are included, according to additional alternative embodiments. ) are used to allow influencing or controlling the properties of state information generated and placed in the process pool. In this system example, a checker program for device C may, for example, request a program (eg, program P-A, program P-B, etc.) to dump more state than normal into the pool at a single moment or for a certain duration. Or, heralding the next "level" of debug communication, an interested program can request that a program (e.g., program P-A, program P-B, etc.) emit proteins that enumerate objects existing in its runtime environment, which individually can and can be used via The debug pool interacts. From this it follows that a program of interest can "address" individuals among objects during program runtime, placing proteins in a pool of processes that a particular object will individually occupy and respond to. For example, a program of interest may request an object to emit a reporter protein describing the instantaneous values of all its constituent variables. Even more importantly, a program of interest can change its behavior or the value of its variables via other protein instruction objects.

More specifically, in this example, device C's inspection application places a request (in the form of a protein) for a linked list of objects (eg, "request-object-list") into the pool, which is then processed by each Device (e.g. Device A, Device B, etc.) extraction. In response to this request, each device (e.g., device A, device B, etc.) places into the pool proteins (e.g., protein 1A, protein 1B, etc.) that enumerate objects existing in its runtime environment, which are individually capable and available for Interact via the debug pool.

It follows that, via enumeration from the device, and in response to the enumeration of objects, the inspection application of device C addresses individuals among the objects at program runtime, placing proteins in the pool of processes that a particular object will individually occupy and respond to middle. For example, an inspection application of device C may place a requesting protein (e.g., protein "request report P-A-O", "request report P-B-O") at a point where an object (e.g., object P-A-O, object P-B-O, respectively) emits instantaneous values describing all of its constituent variables. In a pool of reporter proteins (eg, protein 2A, protein 2B, etc.). Each object (e.g. object P-A-O, object P-B-O) extracts its request (e.g. protein "request report P-A-O", "request report P-B-O", respectively) and, in response thereto, will include the requested reported protein (e.g. protein 2A, protein 2B) were placed in the pool. Device C then extracts the individual reporting proteins (eg, protein 2A, protein 2B, etc.) and takes subsequent processing actions appropriate to the reported content.

In this way, using plasma as a medium of exchange ultimately tends to erode the distinction between debugging, process control, and program-to-program communication and coordination.

So far, the generic plasma framework allows the design of visualization and analysis programs in a loosely coupled manner. For example, a visualizer that shows memory access patterns can be used in conjunction with any program that outputs its basic memory reads and writes to the pool. The program being profiled does not need to know about the existence or design of the visualization tool, and vice versa.

Using pools in the manner described above does not unduly impact system performance. For example, embodiments have allowed hundreds of thousands of proteins per second to be pooled, enabling even relatively lengthy data output without significantly inhibiting the responsiveness or interactive nature of most programs.

Space Operating Environment (SOE).

The multi-process interactive system may be a component of or coupled for a space operating environment (SOE). An SOE including an attitude control system or an attitude-based control system may also be referred to as a Space User Interface (SUI) or a Space Interface (SI). As an example, Figure 16 is a block diagram of a space operating environment (SOE), under an embodiment. The user positions his hands 1601 and 1602 in the viewing area 1650 of the array of cameras 1604A-1604D. The camera detects the position, orientation, and movement of fingers and hands 1601 and 1602 and generates an output signal to preprocessor 1605 . The pre-processor 1605 translates the camera output into gesture signals provided to the computer processing unit 1607 of the system. Computer 1607 uses the input information to generate commands for controlling one or more cursors on the screen and provides video output to display 1603 .

While the system shown uses a single user's hand as input, multiple users can be used to implement SOE. Furthermore, instead of or in addition to hands, the system may track any one or more parts of the user's body, including head, feet, legs, arms, elbows, knees, and the like.

In the illustrated embodiment, four cameras or sensors are used to detect the position, orientation, and movement of the user's hands 1601 and 1602 in the viewing area 1650 . It should be understood that an SOE may include more (eg, six cameras, eight cameras, etc.) or fewer (eg, two cameras) cameras or sensors without departing from the scope and spirit of the SOE. Furthermore, although the cameras or sensors are arranged symmetrically in example embodiments, there is no requirement for such symmetry in the SOE. Any number of cameras or sensors or positioning of cameras or sensors that allow for the position, orientation and movement of the user's hand may be used in the SOE.

In one embodiment, the camera used is a motion capture camera capable of capturing grayscale images. In one embodiment, the camera used is a camera manufactured by Vicon (eg, a Vicon MX40 camera). The camera includes on-camera processing and is capable of image capture at 1000 frames per second. Motion capture cameras are able to detect and locate markers.

In the described embodiment, the camera is a sensor for optical detection. In other embodiments, cameras or other detectors may be used for electromagnetic, magnetostatic, RFID, or any other suitable type of detection.

The preprocessor 1605 generates 3D space point reconstruction and skeleton point annotation. A gesture translator 1606 converts the 3D spatial information and marker motion information into a command language that the computer processor can interpret to update the position, shape and motion of the cursor on the display. In an alternative embodiment of the SOE, the pre-processor 1605 and gesture translator 1606 are integrated or combined into a single device.

Computer 1607 may be any general purpose computer made by, for example, Apple, Dell, or any other suitable manufacturer. Computer 1607 runs application programs and provides display output. Cursor information that would otherwise come from a mouse or other prior art input device now comes from the gesture system.

The SOE or embodiment contemplates the use of a mark tag on one or more of the user's fingers so that the system can locate the user's hand, identify whether it is looking at the left or right hand, and which fingers are visible. This allows the system to detect the position, orientation and movement of the user's hand. This information allows multiple gestures to be recognized by the system and used as commands by the user.

The marker tag in one embodiment is a physical tag comprising (in this embodiment adapted to be attached at various locations on a human hand) a substrate and discrete indicia disposed on the surface of the substrate in a uniquely identifying pattern.

The markers and associated external sensing systems can operate in any domain (optical, electromagnetic, magnetostatic, etc.) that allows precise, accurate and rapid as well as continuous acquisition of their three-space position. The markers themselves may operate actively (eg by emitting structured electromagnetic pulses) or passively (eg, in this embodiment, by optically reflecting light).

At each frame acquired, the detection system receives a converging "cloud" comprising the recovered three-space positions of all markers from the markers currently in the instrumented workspace volume (within view of the camera or other detector). The markers on each label are sufficiently diverse and arranged in a unique pattern that a detection system can perform the following tasks: (1) Segmentation, where each recovered marker position is assigned to a point forming a single label One and only a subset of ; (2) labeling, where a subset of each segment of points is identified as a specific label; (3) location, where the three-space position of the identified label is recovered; (4) orientation , where the three-space orientation of the identified label is recovered. As described below, and as shown in one embodiment in FIG. 17 , tasks (1 ) and (2) are made possible by certain properties of the marking pattern.

The markings on the label in one embodiment are attached to a subset of the regular grid locations. As in this embodiment, such a potential grid may be of the traditional Cartesian class, or may instead be some other conventional planar tessellation (eg a triangular/hexagonal bedding arrangement). The known spatial resolution of the marker sensing system establishes the scale and space of the grid such that adjacent grid positions cannot be confused. The choice of labeling pattern for all labels should satisfy the constraint that a pattern with no labels will conform to any other labeling pattern by any combination of rotation, translation or mirroring. The diversity and arrangement of markers can further be chosen such that loss (or blocking) of some specified number of component markers is tolerable. After any arbitrary transformations, it should still be impossible to confuse a compromised module with any other.

Referring now to FIG. 17, a plurality of tags 1701A-1701E (left hand) and 1702A-1702E (right hand) are shown. Each label is rectangular and in this embodiment comprises a 5x7 grid array. Pick a rectangular shape as an aid in orienting the label and reduce the possibility of mirror duplication. In the illustrated embodiment, there is a label for each finger on each hand. In some embodiments, it may be sufficient to use one, two, three, or four tags per hand. Each label has a border of a different grayscale or color shade. Within that boundary is a 3x5 grid array. Markers (indicated by black dots in Figure 17) are placed at specific points in the grid array to provide information.

Quality information can be encoded in a tag's tagging schema by segmenting each schema into "common" and "unique" sub-patterns. For example, this embodiment specifies two possible "boundary patterns", distributions of markers about the boundaries of a rectangle. A "family" of tags is thus established - the tags for the left hand can thus all use the same boundary pattern as shown by tags 1701A-1701E, while the tags attached to the fingers of the right hand can be assigned different border patterns as shown by tags 1702A-1702E. model. This subpattern is chosen such that in all orientations of the label, left and right patterns are distinguished. In the illustrated example, the left-handed pattern includes markings on each corner and markings on the second from the corner grid positions. The right hand pattern has marks on only two corners and two marks on non-corner grid positions. Examination of the modes shows that as long as any three of the four markers are visible, one can definitely distinguish left-handed from left-handed. In one embodiment, the color or shading of the border may also be used as an indicator of handedness.

Of course, each tag must still employ a unique internal schema, tags distributed within the common boundaries of the tag's family. In the illustrated embodiment, it has been found that two markers in the inner grid array are sufficient to uniquely identify each of the ten fingers without duplication due to the rotation or orientation of the fingers. Even if one of the tags is blocked, the combination of the tag's handedness and pattern produces a unique identifier.

In this embodiment, the grid location is visually represented on the rigid substrate as an aid to the (manual) task of attaching each reflective marker at its intended location. These grids and expected marking positions are literally printed via a color inkjet printer onto a substrate, here a (initially) sheet of flexible "shrink film". Each module is cut from sheet material and then oven baked, during which heat treatment each module undergoes precise and repeatable shrinkage. For example, for a brief interval after the process, the cooled label can be slightly shaped - to follow the longitudinal curve of the finger; the substrate is then suitably rigid and markings can be affixed at the indicated grid points.

In one embodiment, the markings themselves are three-dimensional (eg, small reflective spheres attached to the substrate via adhesive or some other suitable means). The three-dimensionality of the markers can be an aid in detection and localization on two-dimensional markers. However, either may be used without departing from the spirit and scope of the SOE described herein.

Currently, the tags are attached via Velcro or other suitable means to the glove worn by the operator, or alternatively directly to the operator's finger using a soft double stick strap. In a third embodiment, individual markers can be dispensed along with the rigid substrate and affixed - or "painted" - onto the operator's fingers and hands.

The SOE of an embodiment is expected to include a gesture vocabulary of hand pose, orientation, hand combination, and orientation mix. A sign language is also implemented for designing and communicating poses and gestures in the SOE's gesture vocabulary. A pose vocabulary is a system for representing instantaneous "pose states" of kinematic linkages in a compact textual form. The linkages in question may be biological (such as a human hand; or an entire human body, or a locust leg, or the articulated spine of a lemur), or may instead be non-biological (such as a robotic arm). In any case, links can be simple (spine) or branched (hands). SOE's gesture vocabulary system builds a constant-length string for any particular link, and the aggregation of specific ASCII characters occupying a "character position" of the string is the unique description of the link's "pose" or immediate state.

Figure 18 illustrates hand gestures in an example of the gesture vocabulary of the SOE, according to an embodiment. SOE assumes use of each of the five fingers on the hand. These fingers are coded as p-little, r-ring, m-middle, i-index, and t-thumb. Figure 18 defines and illustrates a number of gestures for fingers and thumbs. The gesture vocabulary string establishes a single character position for each expressible degree of freedom of the link (in this case, a finger). Furthermore, each such degree of freedom is understood to be discretized (or "quantized") such that its full range of motion can be expressed by assigning one of a finite number of standard ASCII characters at a string position. These degrees of freedom are expressed relative to a body-specific origin (back of hand, center of locust body, base of robot arm, etc.) and coordinate system. Thus, a small number of additional gesture-lexical character positions are used to express the position and orientation of links "in general" in a more global coordinate system.

Still referring to FIG. 18, a number of gestures are defined and identified using ASCII characters. Distinguish some gestures between thumb and non-thumb. The SOE in this embodiment uses an encoding such that the ASCII characters themselves are suggestive of gestures. However, any character may be used to indicate a gesture, implied or not. Furthermore, it is not required to use ASCII characters for symbol strings in the present invention. Any suitable symbols, numbers or other representations may be used without departing from the scope and spirit of the invention. For example, a symbol may use two bits per finger, if desired, or some other number of bits, as desired.

The characters "^" indicate a bent finger, and ">" indicate a bent thumb. A straight finger or thumb pointing upward is indicated by a "1" and at an angle is indicated by a "\" or "/". A "-" indicates a thumb pointing straight across, and an "x" indicates a thumb pointing flat.

Using these individual finger and thumb descriptions, a robust number of hand poses can be defined and programmed using the scheme of the present invention. Each pose is represented by five characters in the order p-r-m-i-t as described above. Figure 18 shows a number of gestures, and some are described here by way of illustration and example. "11111" indicates a hand kept flat and parallel to the ground. "^^^^>" means fist. "111^>" represents the "OK" symbol.

Strings offer the opportunity for immediate "human readability" when implied characters are used. The set of possible characters describing each degree of freedom can usually be picked by eye for quick identification and obvious analogy. For example, a vertical bar ("|") would likely indicate that a link element is "straight", a right-angle bend ("L") could indicate a ninety-degree bend, and a circumflex ("^") could indicate a sharp bend. As mentioned above, any character or encoding can be used as desired.

Any system employing pose vocabulary strings as described herein enjoys the advantage of the high computational efficiency of string comparisons - the identification or search for any given pose becomes entirely a "string comparison" between the desired pose string and the instantaneous actual string (eg UNIX "strcmp()" function). Furthermore, the use of wildcard characters provides programmers or system designers with additional familiar efficiency and efficacy: degrees of freedom whose immediate state is irrelevant for matching can be designated as question mark points ("?"); additional wildcard meanings can be assigned.

In addition to finger and thumb gestures, hand orientation can represent information. Characters describing directions in global space can obviously also be chosen: the characters "<", ">", "^" and "v" when encountered in a direction character position can be used to indicate left, right, up and down idea. Figure 19 shows an encoded hand orientation descriptor and an example combining pose and orientation. In an embodiment, the two character positions first specify the orientation of the palm, then the orientation of the fingers (whether they are straight or not, regardless of the actual curvature of the fingers). Possible characters for these two positions represent the "center of body" symbols for orientation: "-", "+", "x", "*", "^", and "v" describe middle, side, front ( forward, away from the body), rear (backward, away from the body), head (up) and tail (down).

In the notation scheme of an embodiment of the invention, a five finger gesture indicating a character is followed by a colon and then two directional characters to define a complete command gesture. In one embodiment, the starting position is referred to as the "xyz" pose, where the thumb is pointing straight up, the index finger is pointing forward, and the middle finger is perpendicular to the index finger, pointing left when performing this pose with the right hand. This is represented by the string "^^x1-:-x".

"XYZ-Hand" is a technique that exploits the geometry of the human hand to allow full six degrees of freedom navigation of visually rendered three-dimensional structures. While the technique relies only on massive translation and rotation of the operator's hand - so that his fingers can be held in essentially any pose desired - this embodiment prefers a static configuration with the index finger pointing away from the body and the thumb pointing toward the ceiling ; The middle finger points left and right. The three fingers thus describe (roughly, but with a clear and obvious intent) the three mutually orthogonal axes of a three-dimensional coordinate system: hence "XYZ-hand".

XYZ-hand navigation is then performed with the hand, fingers in the above-mentioned posture held in front of the operator's body at a predetermined "neutral position". Access to the three translational and three rotational degrees of freedom of a three-space object (or camera) is affected in the following natural ways: Left-right movement of the hand (relative to the body's natural coordinate system) produces movement along the computed context's x-axis , up and down movement of the hand produces movement along the y-axis of the controlled context, and forward and backward hand movement (towards/away from the operator's body) produces z-axis movement within the context. Similarly, rotation of the operator's hand about the index finger results in a "roll" change in the orientation of the computed context; "tilt" and "yaw" changes are similarly affected by rotation of the operator's hand about the middle finger and thumb, respectively.

Note that while "computational context" is used here to denote entities controlled by an XYZ-hand approach—and seem to suggest synthetic three-dimensional objects or cameras—it should be understood that this technique is useful for controlling real-world objects. Various degrees of freedom: pan/tilt/roll control of a video or motion picture camera equipped with appropriate rotational actuators is equally useful. Furthermore, even in the virtual domain, the body degrees of freedom afforded by XYZ-hand poses can be mapped somewhat less literally. In this embodiment, the XYZ-hands are also used to provide navigational access to the large panorama display image, such that side-to-side and up-down movements of the operator's hand result in a desired side-to-side or up-down "pan" about the image, but the operator's The back and forth movement of the hand is mapped as a "zoom" control.

In each case, the coupling between hand motion and the resulting computational translation/rotation can be direct (i.e. operator's hand position or rotational offset is mapped one-to-one to computational position or rotation offset of the camera or object in the context of the computation) or indirect (i.e. the position or rotation offset of the operator's hand is mapped one-to-one to the position/orientation in the computed context via some linear or non-linear function order or higher derivatives, the ongoing integration then acts on a non-stationary change of the actual zero-order position/orientation of the context of the computation). This latter means of control is similar to the use of a car "gas pedal", where a constant deflection of the pedal, more or less, results in a constant vehicle speed.

A "neutral position" that serves as the origin of the real-world XYZ-hand's local 6DOF coordinates can be established as (1) an absolute position and orientation in space (such as relative to an enclosed room); (2) relative to the operator's own Fixed position and orientation (e.g., eight inches in front of the body, ten inches below the chin, sideways at shoulder level), regardless of the operator's general position and "heading," or (3) interactively, by the operator Deliberate secondary actions of , (e.g., gesture commands issued using the operator's "other" hand indicating that the current position and orientation of the XYZ-hand is used from here on as the origin of translation and rotation).

It is further convenient to provide a "detent" region (or "dead zone") about the XYZ-hand neutral position, so that motion within this volume does not map to motion in the context of the control.

Other poses can include:

[|||||:vx] is a straight hand with the palm facing down and the fingers forward (thumb parallel to the fingers).

[|||||:x^] is a straight hand with the palm facing forward and the fingers facing the ceiling.

[|||||:-x] is a straight hand with the palm facing the center of the body (right if left handed, left if right handed) and fingers forward

[^^^^-:-x] is one hand with the thumb up (with the thumb pointing toward the ceiling).

[^^^|-:-x] refers to a forward imitation pistol.

The SOE of an embodiment anticipates one-handed commands and gestures as well as two-handed commands and gestures. Figure 20 shows an example of two-hand combinations and associated symbols in an embodiment of the SOE. Referring back to the notation of the first example, "period" indicates that it consists of two closed fists. The "snapshot" example has the thumb and forefinger of each hand extended, with the thumbs pointing towards each other, defining a goalpost shaped frame. "Rudder and Throttle Start Position" is fingers and thumb pointing up with the palm facing the screen.

Figure 21 shows an example of directional mixing in an embodiment of the SOE. In the example shown, blending is indicated by pairs of directional symbols enclosed in parentheses following the string of finger gestures. For example, the first command shows all pointed finger positions. The first pair of directional commands will produce a palm facing straight toward the display, the second pair has fingers rotated to tilt 45 degrees toward the screen. While multiple pairs of mixes are shown in this example, any number of mixes are contemplated in the SOE.

Figure 23 shows a number of possible commands that can be used for SOE. While there has been some discussion herein regarding controlling a cursor on a display, SOE is not limited to this activity. In fact, SOE has great application in manipulating any and all data on the screen, portions of data, and the state of the display. For example, commands may be used to override video controls during playback of video media. Commands can be used to pause, fast forward, rewind, etc. In addition, the commands can be implemented as zooming in or out of the image, changing the orientation of the image, panning in any direction, etc. SOE can also be used in place of menu commands such as Open, Close, Save, etc. In other words, any command or activity imaginable can be achieved through hand gestures.

Figure 22 is a flowchart of the operation of the SOE according to an embodiment. At 2201, a detection system detects markers and labels. At 2202, it is determined whether tags and markers are detected. If not, the system returns 2201. If tags and markers are detected at 2202, the system proceeds to 2203. At 2203, the system recognizes hands, fingers, and gestures from the detected tags and markers. At 2204, the system identifies the direction of the gesture. At 2205, the system identifies the three-dimensional spatial location of the detected hand or hands (note that any or all of 2203, 2204, and 2205 can be combined).

At 2206, the information is translated into gesture symbols as described above. At 2207, it is determined whether the gesture is valid. This can be done via simple string comparison using the generated string of symbols. If the gesture is invalid, the system returns to 2201. If the gesture is valid, then at 2208 the system sends symbol and location information to the computer. At 2209, the computer determines the appropriate action to perform in response to the gesture, and updates the display at 2210 accordingly.

In one embodiment of the SOE, operations 2201-2205 are performed by an on-camera processor. In other embodiments, processing may be accomplished by the system computer, if desired.

The system is capable of "parsing" and "translating" the stream of low-level gestures recovered by the underlying system, and turning these parsed and translated gestures into commands or events that can be used to control a wide range of computer applications and systems flow of data. These techniques and algorithms can be embodied in systems that include computer code that provides an engine that implements these techniques and a platform for building computer applications that exploit the capabilities of the engine.

One embodiment focuses on enabling rich gestures using the human hand in computer interfaces, but is also capable of recognizing gestures made by other body parts including, but not limited to, arms, torso, legs, and head, as well as static and articulated various non-hand physical tools , including but not limited to calipers, compasses, flexible curve approximators, and pointing devices of various shapes. Markings and labels can be applied as desired to items and tools carried and used by operators.

The system described here incorporates several innovations that make it possible to build gesture systems rich in the range of gestures that can be recognized and acted upon, while being easy to integrate into applications.

The gesture parsing and translation system in one embodiment includes:

1) A concise and efficient way to specify (code for use in a computer program) poses for several different levels of convergence:

a. The "pose" (position and orientation of parts of the hand relative to each other) of the hand in three-dimensional space, the orientation and position of the hand.

b. Two-handed combination, one hand used to account for posture, position, or both

c. Multiplayer teaming; the system can track more than two hands, so more than one person can collaboratively (or in the case of gaming applications, competitively) control the target system.

d. Sequential poses in which poses are combined sequentially, we call "animating" poses.

e. "Glyph" poses, where the operator traces a shape in space

2) Programming techniques for registering specific gestures from each of the above categories relevant to a given application context

3) Algorithms for parsing the gesture stream so that registered gestures can be recognized and events encapsulating these gestures can be delivered to the relevant application context.

The canonical system (1) with constituent elements (1a) to (1f) provides the basis for using the parsing and translation capabilities of the system described herein.

A one-hand "pose" is represented as the following string:

i) the relative orientation between the fingers and the back of the hand,

ii) Quantization to a few discrete states.

Using relative bonding directions allows the system described here to avoid problems associated with different hand sizes and geometries. The system does not require "operator calibration". Furthermore, specifying poses as strings or sets of relative orientations allows more complex pose specifications to be easily created by combining pose representations with other filters and norms.

Using a small number of discrete states for pose specifications makes it possible to specify poses succinctly and ensures accurate pose recognition using a variety of potential tracking techniques (e.g., passive optical tracking using cameras, active optical tracking using points of light and cameras, electromagnetic field tracking, etc.).

The poses in each category (1a) to (1f) can be specified partially (or minimally), ignoring non-critical data. For example, gestures in which the positions of two fingers are fixed and the positions of the other fingers are unimportant can be made by listing a "wildcard" or a generic "ignore these " designator to represent a single specification.

All innovations described here for pose recognition, including but not limited to multi-layer specification techniques, use of relative orientations, quantification of data and allowing partial or minimal specification at each level, use of other body parts and "fabricated" tools and objects generalize the specification of hand poses to the specification of gestures.

The programming technique for "registering gestures" (2) consists of a set of defined application programming interface calls that allow the programmer to define which gestures the engine should make available to the rest of the runtime system.

These API procedures can be used at application build time to create static interface definitions that are used throughout the life of the running application. They can also be used during the running process, allowing interface properties to change on the fly. This real-time change of the interface makes it possible to:

i) Construct complex context and conditional control state,

ii) dynamically adding hysterism to the control environment, and

iii) Create applications in which the user can change or extend the interface vocabulary of the runtime system itself.

The algorithm for parsing the pose stream (3) compares the pose specified in (1) and registered in (2) with the incoming low-level pose data. When a match for a registered gesture is identified, event data representing the matched gesture is passed on the stack to the running application.

Efficient real-time matching is desired in the design of this system, and specified poses are viewed as trees of probabilities that are processed as quickly as possible.

Furthermore, the native comparison operators used internally to recognize specified gestures are also exposed for use by application programmers, so that another comparison (e.g., flexible state checks in complex or hybrid gestures) can be performed even from the application Appears in context.

Recognizing "lock" semantics is an innovation of the system described here. The Registry API (2) implies these semantics (and, for fewer extensions, embeds within the specification vocabulary (1)). Registration API calls include:

i) "entry" status notifier and "continue" status notifier, and

ii) Gesture priority designator.

If a gesture has been recognized, its "continue" condition takes precedence over all "login" conditions for gestures with the same or lower priority. This distinction between the login state and the continue state adds significantly to the perceived system usability.

The system described herein includes algorithms for robust operation in the face of real world data errors and uncertainties. Data from low-level tracking systems can be incomplete (for various reasons including blockage of markers in optical tracking, network dropout or processing lag, etc.).

Depending on the amount and context of missing data, missing data is flagged by the parsing system and interpolated into

The "last known" or "most probable" state.

If data about a particular pose component (e.g., the orientation of a particular joint point) is missing, but it is physically possible to analyze the "last known" state of that particular component, then the system uses that last known state in its real-time matching .

Conversely, if it is not physically possible to analyze the last known state, the system falls back to a "best guess range" of components and uses this synthetic data in its real-time matching.

The specification and parsing system described here has been carefully designed to support "handedness agnosticism", such that for multi-handed poses, either hand is allowed to satisfy the pose requirements.

The system of an embodiment may provide a virtual space depicted on one or more display devices wherein

("Screen") is considered an environment consistent with the physical space occupied by one or more operators of the system. An example of such an environment is described herein. This current embodiment consists of three projector-driven screens at fixed locations, driven by a single desktop computer, and controlled using the gesture vocabulary and interface system described herein. Note, however, that the technology described supports any number of screens; the screens can be mobile (rather than fixed), the screens can be driven simultaneously by many independent computers, and any input device or technology can control the overall system.

The interface system described in this publication shall have means to determine the size, orientation and position of the screen in physical space. Given this information, the system is able to dynamically map the physical space in which these screens are positioned (and prohibited by the system's operator) to be projected to the virtual space of the computer applications running on the system. As part of this automatic mapping, the system also translates the scale, angle, depth, size and other spatial characteristics of the two spaces in various ways, depending on the needs of the system-hosting application.

This continuous translation between physical and virtual spaces enables the alignment and integration of multiple interface technologies that are difficult to implement on existing application platforms or must be implemented piece-meal for each application running on existing platforms. Universal use is possible. These techniques include (but are not limited to):

1) The use of "literal pointing" - using the hand in a gestural interface environment, or using a physical pointing tool or device - as a pervasive and natural interface technique

2) Automatic compensation for screen motion or repositioning.

3) Graphics rendering that changes based on operator position, such as simulated parallax transitions to enhance depth perception.

4) Inclusion of physical objects in on-screen display - taking into account real world position, orientation, state, etc. For example, an operator standing in front of a large transparent screen can see application graphics and representations of the scale model's true location (and perhaps moving or changing direction) behind the screen.

It is important to note that text pointing is distinct from the abstract pointing used in mouse-based windowing interfaces and most other modern systems. In these systems, the operator must learn to manage the translation between the virtual pointer and the physical pointing device, and must cognitively map between the two.

By contrast, in the system described in this publication, there is no difference between virtual and physical space from the application or user point of view (except that the virtual space is more amenable to mathematical manipulation), so there is no need for the operator to cognitive translation.

The closest analogue to the textual pointers provided by the embodiments described herein are touch-sensitive screens (eg, found on many ATM machines). Touch-sensitive screens provide a one-to-one mapping between the two-dimensional display space on the screen and the two-dimensional input space on the screen surface. In a similar manner, the system described herein provides flexible mapping (possibly, but not necessarily, one-to-one) between the virtual space displayed on one or more screens and the physical space occupied by the operator. Regardless of the usefulness of the analogy, it is worth understanding that the extension of this "mapping method" to three dimensions and to arbitrarily large architectural environments and multiple screens is not trivial.

In addition to the components described here, the system may implement algorithms that enable a continuous system-level mapping (perhaps modified by rotation, translation, scaling, or other geometric transformations) between the physical space of the environment and the display space of each screen.

A rendering stack that takes compute objects and maps and outputs a graphical representation of the virtual space.

An input event processing stack that takes event data from the control system (in the current embodiment, gesture and pointing data from system and mouse input) and maps spatial data from input events to coordinates in virtual space. Translated events are then delivered to the running application.

A "glue layer" that allows a system to host applications running across several computers on a LAN.

As described above with reference to FIGS. 1A-1C and otherwise described herein, considering the above description of the SOE, the SOE may be used as a component of and/or coupled to a multi-process interactive system. As mentioned above, the SOE of an embodiment may be encapsulated as a gesture/spatial process G that transfers user input proteins to the user input pool Ui.

Embodiments herein include systems and methods for detecting gestures made by a body from gesture data. Pose data is received via a detector. The systems and methods of embodiments execute a multitude of processes on a processing device. The process generates an event that includes a collection of events representing gestures. The systems and methods of embodiments translate each process's events into data capsules. The systems and methods of embodiments transfer data capsules to multiple pools or storages. A collection of processes in a large number of processes operates as a recognition process. The recognition process identifies data capsules in the pool that include content corresponding to the pose. The recognition process retrieves the recognized data capsules from the pool and generates a pose signal from the recognized data capsules by synthesizing the contents of the recognized data capsules to form the pose signal. The attitude signal indicates attitude.

Figure 24 is a block diagram of a space operating environment (SOE) (see Figure 1C, element G) implemented by or as a component of a multi-process interactive system, under an embodiment. The user positions his hands 2401 and 2402 in the viewing area 2450 of the array of cameras 2404A-2404D. The camera detects the position, orientation, and movement of fingers and hands 2401 and 2402 and generates an output signal to preprocessor 2405. The pre-processor 2405 translates the camera output into pose signals that are provided to the system's computer processor. In this embodiment, the functions of the computer processor performed by the computer 2407 described above may be performed by a processor of the multi-process interactive system and/or a processor coupled to the multi-process interactive system (FIG. 1C). The gesture signal can be provided or transmitted to the pool of the multi-process interactive system (pool Ui, Fig. 1C). Accordingly, the multi-process interactive system uses gesture signals to generate commands for controlling one or more components coupled to the multi-process interactive system (eg, a display cursor, etc.).

Although the system is shown with a single user's hand as input, multiple users can be used to implement SOE. Furthermore, instead of or in addition to hands, the system may track any one or more parts of the user's body, including head, feet, legs, arms, elbows, knees, and the like.

In the illustrated embodiment, four cameras or sensors are used to detect the position, orientation, and movement of the user's hands 2401 and 2402 in the viewing area 2450 . It should be understood that an SOE may include more (eg, six cameras, eight cameras, etc.) or fewer (eg, two cameras) cameras or sensors without departing from the scope and spirit of the SOE. Furthermore, although the cameras or sensors are arranged symmetrically in example embodiments, there is no requirement for such symmetry in the SOE. Any number of cameras or sensors or positioning of cameras or sensors that allow for the position, orientation and movement of the user's hand may be used in the SOE.

In one embodiment, the camera used is a motion capture camera capable of capturing grayscale images. In one embodiment, the camera used is a camera manufactured by Vicon (eg, a Vicon MX40 camera). The camera includes on-camera processing and is capable of image capture at 1000 frames per second. Motion capture cameras are able to detect and locate markers.

In the described embodiment, the camera is a sensor for optical detection. In other embodiments, cameras or other detectors may be used for electromagnetic, magnetostatic, RFID, or any other suitable type of detection.

The preprocessor 2405 generates 3D space point reconstruction and skeleton point annotation. The pose translator 2406 converts the 3D spatial information and marker motion information into a command language that can be interpreted by the components of the multi-process interaction system receiving the information from the pool Ui (see FIG. 1 ). In an alternative embodiment of the SOE, the pre-processor 2405 and gesture translator 106 are integrated or combined into a single device.

25 is a flowchart 2500 of the operation of the multi-process interaction system 100 using input from a gesture control system, under an embodiment. Operations include detecting 2502 gestures performed by the body from the gesture data. Pose data is received via a detector. Operations include executing a number of processes on the processing device 2504 . The process generates an event that includes a collection of events representing gestures. A process includes, but is not limited to, a separable program execution context for a spatial manipulation application. Events for each process are translated into data capsules 2506 . A data capsule includes, but is not limited to, an application-independent representation of event data for events and state information of the process that initiated the data capsule. The data capsules are translated into pools 2508. A collection of processes of a large number of processes operates as an identifying process. The recognition process identifies a data capsule 2510 in the pool that includes content corresponding to the gesture. The recognition process retrieves the recognized data capsules from the pool and generates a pose signal 2512 from the recognized data capsules by synthesizing the contents of the recognized data capsules to form the pose signal. The attitude signal indicates attitude.

Embodiments described herein include a method comprising: executing a plurality of processes on at least one processing device; translating events of each of the plurality of processes into data capsules, transmitting the data capsules to a plurality of pools; a process operating as a recognition process that identifies, in a plurality of pools, a data capsule comprising at least one of content corresponding to an interactive function of the recognition process and an identification of the recognition process; and the recognition process retrieves the recognized data from the plurality of pools capsule, and perform appropriate processing on the contents of the identified data capsule.

The data capsule of an embodiment includes an application-independent representation of the state information of the process originating the data message and the event data of the event.

The method of an embodiment includes forming an interactive application from the plurality of processes by coordinating the operation of each of the plurality of processes using the data capsule and the plurality of pools.

The method of an embodiment includes coordinating operations of the plurality of processes using at least one of the data capsule and the plurality of pools.

The method of an embodiment includes: dividing an application program into a set of processes, wherein the plurality of processes includes a set of processes.

The method of an embodiment includes the process of generating an output by interactively processing a plurality of retrieved data capsules of at least one of the plurality of pools.

The multiple processes of an embodiment comprise separable program execution contexts of multiple application programs, wherein each application program comprises at least one process.

The method of an embodiment includes: executing multiple processes in parallel.

The method of an embodiment includes executing a first set of processes in parallel and executing a second set of processes sequentially, wherein the plurality of processes includes the first set of processes and the second set of processes.

An event of an embodiment represents process input.

An event of an embodiment represents process output.

The events of an embodiment include user interface events.

The events of an embodiment include graphics events.

An event of an embodiment represents a process state.

The state of the process of the embodiment represents the interactive function of the process, wherein the interactive function of the process is exposed to multiple processes as the content of the data capsule.

The method of an embodiment includes: defining an application programming interface (API) of a plurality of processes through the content of the data capsule, rather than defining the API through function calls.

The content of the data capsule of an embodiment is application independent and identifiable by multiple processes.

The at least one processing device of an embodiment comprises a plurality of processing devices.

A first set of at least one process in the plurality of processes of the embodiment runs under the first set of at least one processing device in the plurality of processing devices, and a second set of at least one process in the plurality of processes runs on the plurality of processing devices The at least one processing device in the second set operates.

The plurality of processes of an embodiment includes a first process.

The translating of an embodiment includes transforming an event of the first process into at least one data sequence comprising first process event data specifying the event and state information of the event.

The first process event data and state information of an embodiment are type-specific data having a type corresponding to an application program of the first process.

The translating of an embodiment includes forming a data capsule to include at least one data sequence, the data capsule having a data structure including an application-independent representation of the at least one data sequence.

The plurality of processes of an embodiment includes a second process.

The translating of an embodiment includes transforming the state change event of the second process into at least one data sequence comprising second process event data specifying the event and state information of the event.

The second process event data and state information of an embodiment are type-specific data having a type corresponding to an application program of the second process.

The translating of an embodiment includes forming a data capsule to include at least one data sequence, the data capsule having a data structure including an application-independent representation of the at least one data sequence.

The identification process of an embodiment is a second process, and the retrieval includes retrieving the identified data capsules from the plurality of pools and performing the second process of processing appropriate to the contents of the identified data capsules.

The content of the identified data capsule of an embodiment is data representing state information of the first process.

The translating of an embodiment includes transforming the content of the identified data capsule into at least one new data sequence representing at least one of an event of the first process and an event of the second process.

The at least one new data sequence of an embodiment includes state information of at least one of the first process and the second process and event data specifying an event.

The event data and state information of at least one of the first process and the second process of an embodiment are type-specific data having a type corresponding to an application program of the at least one of the first process and the second process.

The translating of an embodiment includes forming a data capsule to include at least one new data sequence, the data capsule having a data structure that includes an application-independent representation of the at least one new data sequence.

Processes of an embodiment use at least one new data sequence.

Proper processing of the content of the identified data capsule of an embodiment includes rendering the graphical object, wherein the graphical object is rendered on a display of the at least one processing device.

The rendering of an embodiment includes direct rendering, wherein multiple processes draw directly to a graphics layer of at least one processing device, wherein multiple pools are used for coordination among the multiple processes as appropriate for rendering.

The rendering of an embodiment includes transferring a data capsule comprising rendering commands to a plurality of processes of a plurality of pools. The rendering of an embodiment includes retrieving rendering commands from a plurality of pools, interpreting the rendering commands, and driving a plurality of processes of a graphics layer of at least one processing device in response to the rendering commands.

The rendering of an embodiment includes multiple processes rendering to pixel buffers. The rendering of an embodiment includes: a plurality of processes passing raw frame data resulting from rendering to a pixel buffer to a plurality of pools. The rendering of an embodiment includes retrieving raw frame data from a plurality of pools and combining said raw frame data for use in a plurality of processes driving a graphics layer of at least one processing device.

The method of an embodiment includes detecting events of a plurality of processes. The method of an embodiment includes generating at least one data sequence comprising event data for a specified event and state information for the event, wherein the event data and state information are type-specific with a type corresponding to an application program of at least one processing device data. The method of an embodiment comprises forming a data capsule comprising at least one data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one data sequence.

The generating at least one data sequence of an embodiment comprises generating a first respective data set comprising first respective event data. The generating of at least one data sequence of an embodiment comprises generating a second respective data set comprising second respective state information. The generating of the at least one data sequence of an embodiment comprises forming the first data sequence to include a first respective data set and a second respective data set.

Generating the first respective data set of an embodiment comprises forming the first respective data set to include identification data of the at least one processing device, the identification data comprising data identifying the at least one processing device.

The generating at least one data sequence of an embodiment comprises generating a first respective data set comprising first respective event data. The generating of at least one data sequence of an embodiment comprises generating a second respective data set comprising second respective state information. The generating of the at least one data sequence of an embodiment comprises forming the second data sequence to include the first respective data set and the second respective data set.

The generating the first respective data set of an embodiment comprises: generating the first respective data set offset, wherein the first respective data set offset points to the first respective data set of the second data sequence.

The generating of the second respective data set of an embodiment comprises: generating a second respective data set offset, wherein the second respective data set offset points to a second respective data set of the second data sequence.

The first respective data set of an embodiment is a description list that includes a description of the data.

The event data of an embodiment is a tagged sequence of bytes representing typed data.

The event data of an embodiment includes a type header and a type-specific data layout.

The state information of an embodiment is a tagged sequence of bytes representing typed data.

The state information of an embodiment includes a type header and a type-specific data layout.

The method of an embodiment includes generating at least one offset. The method of an embodiment includes forming the data capsule to include at least one offset.

The method of an embodiment includes generating a first offset having a first variable length, wherein the first offset points to event data of a first data sequence of the at least one data sequence.

The method of an embodiment includes: generating a second offset having a second variable length, wherein the second offset points to state information of a first data sequence in the at least one data sequence.

The method of an embodiment includes forming a first code path through the data capsule using a first offset of the at least one offset. The method of an embodiment includes forming a second code path through the data capsule using a second offset of the at least one offset, wherein the first code path and the second code path are different paths.

At least one of the first offset and the second offset of an embodiment includes metadata including context specific metadata corresponding to a context of the application.

The method of an embodiment includes generating a header including a length of the data capsule. The method of an embodiment includes forming the data capsule to include the header.

The method of an embodiment includes transmitting the data capsule to a pool of the plurality of pools.

The method of an embodiment includes detecting a second event of at least one processing device. The method of an embodiment includes searching the plurality of pools for a data capsule corresponding to the second event.

The method of an embodiment includes identifying a correspondence between the data capsule and the second event. The method of an embodiment includes extracting a data capsule from the pool in response to identifying. The method of an embodiment includes: in response to the content of the data capsule, performing a processing operation corresponding to the second event on behalf of at least one processing device, wherein the at least one processing device is associated with the application program of the first type and the second application program of the second type Program correspondence.

The plurality of pools of an embodiment is coupled to a plurality of applications, the plurality of pools includes a plurality of data capsules corresponding to the plurality of applications, the plurality of pools provides access to the plurality of data capsules through the plurality of applications, wherein the plurality of At least two of the applications are different applications.

The multiple pools of an embodiment provide a stateful cache of multiple data capsules.

The multiple pools of an embodiment provide a linear ordering of multiple data capsules.

The data structures of an embodiment are untyped.

The data structure of the data capsule of an embodiment provides a platform-independent representation of event data and state information.

The data structure of the data capsule of an embodiment provides platform-independent access to event data and state information.

The transferring of an embodiment includes transferring the data capsule from a first application having a first application type to at least one second application having at least one second application type, wherein the first application type is the same as the second application Depending on the type of program, wherein generating the at least one data sequence is performed by a first application, the method includes keeping intact the at least one data sequence of the data capsule during the transfer.

The method of an embodiment includes using at least one sequence of data during operation of the second application.

The method of an embodiment includes generating a first data set comprising event data for a source device of at least one processing device, the device event data including data specifying events registered by the source device, and identification data including data identifying the source device.

The method of an embodiment includes generating a second data set comprising the complete set of state information for the event, wherein each of the first data set and the second data set comprises a typed data bundle in a type-specific data layout.

The translating of an embodiment includes encapsulating the first data set and the second data set by forming a data capsule to include the first data set and the second data set, wherein the data capsule has an application-independent Represented data structure.

The method of an embodiment includes detecting an event of a first processing device running under an application of a first type. The method of an embodiment includes generating a data sequence comprising event data of the first processing device, the event data specifying the event and state information for the event, wherein the event data and state information are type-specific data having a type corresponding to the application. The method of an embodiment includes forming a data capsule to include a data sequence, the data capsule having a data structure including an application-independent representation of the data sequence. The method of an embodiment comprises: detecting a second event of a second processing device running under at least one second application of at least one second type, wherein the second type is different from the first type, wherein the at least one processing device A first processing device and a second processing device are included. The method of an embodiment includes identifying a correspondence between the data capsule and the second event. The method of an embodiment comprises: using the contents of the data sequence of the data capsule, performing the operation in response to the second event.

Generating the data sequence of an embodiment includes generating a first data set comprising event data. Generating the data sequence of an embodiment includes generating a second data set including state information. Generating the data sequence of an embodiment includes forming the first data sequence to include the first data set and the second data set.

The event data of an embodiment is a tagged sequence of bytes representing typed data.

The event data of an embodiment includes a type header and a type-specific data layout.

The state information of an embodiment is a tagged sequence of bytes representing typed data.

The state information of an embodiment includes a type header and a type-specific data layout.

The method of an embodiment includes generating at least one offset. The method of an embodiment includes forming the data capsule to include at least one offset.

The method of an embodiment includes generating a first offset having a first variable length, wherein the first offset points to event data of a first data sequence of the at least one data sequence. The method of an embodiment includes: generating a second offset having a second variable length, wherein the second offset points to state information of a first data sequence in the at least one data sequence.

The method of an embodiment includes forming a first code path through the data capsule using a first offset of the at least one offset. The method of an embodiment includes forming a second code path through the data capsule using a second offset of the at least one offset, wherein the first code path and the second code path are different paths.

At least one of the first offset and the second offset of an embodiment includes metadata including context specific metadata corresponding to a context of the application.

The method of an embodiment includes transmitting the data capsule to a pool of the plurality of pools.

The method of an embodiment includes searching the plurality of pools for a data capsule corresponding to the second event. The method of an embodiment includes extracting a data capsule from the pool in response to identifying the correspondence.

The plurality of pools of an embodiment are coupled to the application and at least one second application, the plurality of pools include a plurality of data capsules corresponding to the application and the at least one second application, the plurality of pools provide Two applications have access to multiple data capsules.

The multiple pools of an embodiment provide a stateful cache of multiple data capsules.

The multiple pools of an embodiment provide a linear ordering of multiple data capsules.

The data structures of an embodiment are untyped.

The data structure of the data capsule of an embodiment provides a platform-independent representation of event data and state information.

The data structure of the data capsule of an embodiment provides platform-independent access to event data and state information.

Embodiments described herein include a method comprising: executing a plurality of processes on a processing device, the plurality of processes including separable program execution contexts of a plurality of application programs, wherein each application program includes at least one process; events for each of the plurality of processes are translated into data messages, where the data messages include state information for the process originating the data message and an application-independent representation of the event data for the event; data messages are delivered to multiple pools at least one pool; coordinating among the processes, the coordinating comprising: each process of the plurality of processes coordinating with a peer process of the plurality of processes by retrieving state information of the peer process from the plurality of pools; and Output of the plurality of processes is generated by interactively combining sets of data messages of at least one of the plurality of pools.

Embodiments described herein include a system comprising: at least one processing device that executes a plurality of processes; and a plurality of pools coupled to the at least one processing device; The events of are translated into data capsules, and the data capsules are transmitted to multiple pools; each process of the multiple processes operates as a recognition process, and the recognition process in the multiple pools includes corresponding to the interaction function of the recognition process and the identity of the recognition process a data capsule of at least one of the contents of the identified data capsules; the identification process retrieves the identified data capsules from the plurality of pools, and performs processing appropriate to the contents of the identified data capsules.

Embodiments described herein include a method comprising: executing a plurality of processes on at least one processing device, the plurality of processes comprising separable program execution contexts of a plurality of applications, wherein each application comprises at least one process ; Translate events for each of the multiple processes into data capsules, where the data capsules include state information for the process that originated the data capsule and an application-independent representation of the event data for the event; deliver the data capsule to multiple a pool; each process operates as a recognition process, the recognition process identifies in a plurality of pools a data capsule comprising at least one of content corresponding to an interaction function of the recognition process and an identification of the recognition process; and the recognition process retrieves from the plurality of pools The identified data capsule is identified, and appropriate processing is performed on the contents of the identified data capsule.

The method of an embodiment includes forming an interactive application from the plurality of processes by coordinating the operation of each of the plurality of processes using the data capsule and the plurality of pools.

The method of an embodiment includes coordinating operations of the plurality of processes using at least one of the data capsule and the plurality of pools.

The method of an embodiment includes: dividing an application program into a set of processes, wherein the plurality of processes includes a set of processes.

The method of an embodiment includes the process of generating an output by interactively processing a plurality of retrieved data capsules of at least one of the plurality of pools.

The method of an embodiment includes: executing multiple processes in parallel.

The method of an embodiment includes executing a first set of processes in parallel and executing a second set of processes sequentially, wherein the plurality of processes includes the first set of processes and the second set of processes.

An event of an embodiment represents process input.

An event of an embodiment represents process output.

The events of an embodiment include user interface events.

The events of an embodiment include graphics events.

An event of an embodiment represents a process state.

The state of the process of the embodiment represents the interactive function of the process, wherein the interactive function of the process is exposed to multiple processes as the content of the data capsule.

The method of the embodiment includes: defining the application programming interface APIs of multiple processes through the contents of the data capsules instead of defining the APIs through function calls.

The content of the data capsule of an embodiment is application-independent and recognizable by multiple processes.

The at least one processing device of an embodiment comprises a plurality of processing devices.

A first set of at least one process in the plurality of processes of the embodiment runs under the first set of at least one processing device in the plurality of processing devices, and a second set of at least one process in the plurality of processes runs on the plurality of processing devices The at least one processing device in the second set operates.

The plurality of processes of an embodiment includes a first process.

The translating of an embodiment includes transforming an event of the first process into at least one data sequence comprising first process event data specifying the event and state information of the event.

The first process event data and state information of an embodiment are type-specific data having a type corresponding to an application program of the first process.

The translating of an embodiment includes forming a data capsule to include at least one data sequence, the data capsule having a data structure that includes an application-independent representation of the at least one data sequence.

The plurality of processes of an embodiment includes a second process.

The translating of an embodiment includes transforming the state change event of the second process into at least one data sequence comprising second process event data specifying the event and state information of the event.

The second process event data and state information of an embodiment are type-specific data having a type corresponding to an application program of the second process.

The translating of an embodiment includes forming a data capsule to include at least one data sequence, the data capsule having a data structure that includes an application-independent representation of the at least one data sequence.

The identification process of an embodiment is a second process, and the retrieval includes retrieving the identified data capsules from the plurality of pools and performing the second process of processing appropriate to the contents of the identified data capsules.

The content of the identified data capsule of an embodiment is data representing state information of the first process.

The translating of an embodiment includes transforming the content of the identified data capsule into at least one new data sequence representing at least one of an event of the first process and an event of the second process.

The at least one new data sequence of an embodiment includes state information of at least one of the first process and the second process and event data specifying an event.

The event data and state information of at least one of the first process and the second process of an embodiment are type-specific data having a type corresponding to an application program of the at least one of the first process and the second process.

The translating of an embodiment includes forming a data capsule to include at least one new data sequence, the data capsule having a data structure that includes an application-independent representation of the at least one new data sequence.

Processes of an embodiment use at least one new data sequence.

The processes of an embodiment include an input process that receives input events from input devices.

The translating of an embodiment includes transforming an input event of the input device into at least one data sequence including input device event data specifying the event and state information of the event.

The input device event data and state information of an embodiment are type-specific data having a type corresponding to an application program of the source device.

The translating of an embodiment includes forming a data capsule to include at least one data sequence, the data capsule having a data structure that includes an application-independent representation of the at least one data sequence.

The plurality of processes of an embodiment include pointer processes.

The identification process of an embodiment is a pointer process, and the retrieval includes a pointer process that retrieves the identified data capsules from a plurality of pools and performs appropriate processing on the contents of the identified data capsules.

The content of the identified data capsule of an embodiment is data representing input events from the input process.

The content of the identified data capsule of an embodiment is data representing a position on the display at which a user of the at least one processing device is commanding a pointer object.

The translating of an embodiment includes transforming the content of the identified data capsule into at least one new data sequence, the at least one new data sequence pointing to a defined position of the object with respect to the display.

The at least one new data sequence of an embodiment includes pointer progress event data and pointer progress event status information for a given event.

The pointer process event data and state information of an embodiment are type-specific data having a type corresponding to the application of the pointer process.

The translating of an embodiment includes forming a data capsule to include at least one new data sequence, the data capsule having a data structure that includes an application-independent representation of the at least one new data sequence.

The processes of an embodiment render the pointer object on the display using the at least one new data sequence.

The plurality of processes of an embodiment includes a graphics process.

The translating of an embodiment includes transforming a state change event of a graphics process into at least one data sequence including graphics process event data specifying the event and state information of the event.

The graphics process event data and state information of an embodiment are type-specific data having a type corresponding to an application program of the graphics process.

The translating of an embodiment includes forming a data capsule to include at least one data sequence, the data capsule having a data structure that includes an application-independent representation of the at least one data sequence.

The identification process of an embodiment is a graph process, and the retrieval includes retrieving the identified data capsules from a plurality of pools and performing appropriate processing on the contents of the identified data capsules.

The content of the identified data capsule of an embodiment is data representing state information of another process of the plurality of processes.

The status information of an embodiment includes information of at least one of a space status and a mode status.

The content of the identified data capsule of an embodiment is data representing a position on the display at which a user of the at least one processing device is commanding a pointer object.

The position of the pointer object of an embodiment is within the bounds of the graphics object, wherein the graphics object is rendered by the graphics process.

The translating of an embodiment includes transforming the content of the identified data capsule into at least one new data sequence, the at least one new data sequence representing at least one of a graphic object, a pointer object, an overlapping portion of a pointer object and a boundary.

The at least one new data sequence of an embodiment includes graphics process event data specifying event status information for the graphics process event.

The graphics process event data and state information of an embodiment are type-specific data having a type corresponding to an application program of the graphics process.

The translating of an embodiment includes forming a data capsule to include at least one new data sequence, the data capsule having a data structure that includes an application-independent representation of the at least one new data sequence.

The processes of an embodiment render at least one of the graphics object and the pointer object on the display using the at least one new data sequence.

Appropriate processing of the content of the identified data capsule of an embodiment includes rendering a graphical object, wherein the graphical object is rendered on a display of at least one processing device.

The rendering of an embodiment includes direct rendering, wherein multiple processes draw directly to a graphics layer of at least one processing device, wherein multiple pools are used for coordination among the multiple processes as appropriate for rendering.

The rendering of an embodiment includes transferring a data capsule comprising rendering commands to a plurality of processes of a plurality of pools. The rendering of an embodiment includes retrieving rendering commands from a plurality of pools, interpreting the rendering commands, and driving a plurality of processes of a graphics layer of at least one processing device in response to the rendering commands.

The rendering of an embodiment includes multiple processes rendering to pixel buffers. The rendering of an embodiment includes a plurality of processes that transfer raw frame data resulting from rendering to a pixel buffer to a plurality of pools. The rendering of an embodiment includes retrieving raw frame data from a plurality of pools and combining the raw frame data for use in a plurality of processes driving a graphics layer of at least one processing device.

The method of an embodiment includes detecting events of a plurality of processes. The method of an embodiment comprises generating at least one data sequence comprising event data specifying an event and state information of the event, wherein the event data and state information are type-specific data having a type corresponding to an application of at least one processing device. The method of an embodiment includes forming a data capsule to include at least one data sequence, the data capsule having a data structure that includes an application-independent representation of the at least one data sequence.

The generating at least one data sequence of an embodiment comprises generating a first respective data set comprising first respective event data. The generating of at least one data sequence of an embodiment comprises generating a second respective data set comprising second respective state information. The generating of the at least one data sequence of an embodiment comprises forming the first data sequence to include a first respective data set and a second respective data set.

Generating the first respective data set of an embodiment comprises forming the first respective data set to include identification data of the at least one processing device, the identification data comprising data identifying the at least one processing device.

The generating at least one data sequence of an embodiment comprises generating a first respective data set comprising first respective event data. The generating of the at least one data sequence of an embodiment comprises: generating a second respective data set comprising second respective state information. The generating of the at least one data sequence of an embodiment comprises forming the second data sequence to include the first respective data set and the second respective data set.

The generating the first respective data set of an embodiment comprises: generating the first respective data set offset, wherein the first respective data set offset points to the first respective data set of the second data sequence.

The generating the second respective data set of an embodiment comprises: generating the second respective data set offset, wherein the second respective data set offset points to the second respective data set of the second data sequence.

The first respective data set of an embodiment is a description link list comprising a description of the data.

The event data of an embodiment is a tagged sequence of bytes representing typed data.

The event data of an embodiment includes a type header and a type-specific data layout.

The state information of an embodiment is a tagged sequence of bytes representing typed data.

The state information of an embodiment includes a type header and a type-specific data layout.

The method of an embodiment includes generating at least one offset. The method of an embodiment includes forming the data capsule to include at least one offset.

The method of an embodiment includes generating a first offset having a first variable length, wherein the first offset points to event data of a first data sequence of the at least one data sequence.

The method of an embodiment includes: generating a second offset having a second variable length, wherein the second offset points to state information of a first data sequence in the at least one data sequence.

The method of an embodiment includes forming a first code path through the data capsule using a first offset of the at least one offset. The method of an embodiment includes forming a second code path through the data capsule using a second offset of the at least one offset, wherein the first code path and the second code path are different paths.

At least one of the first offset and the second offset of an embodiment includes metadata including context specific metadata corresponding to a context of the application.

The method of an embodiment includes generating a header including a length of the data capsule. The method of an embodiment includes forming the data capsule to include the header.

The method of an embodiment includes transmitting the data capsule to a pool of the plurality of pools.

The method of an embodiment includes detecting a second event of at least one processing device. The method of an embodiment includes searching the plurality of pools for a data capsule corresponding to the second event.

The method of an embodiment includes identifying a correspondence between the data capsule and the second event.

The method of an embodiment includes extracting a data capsule from the pool in response to identifying. The method of an embodiment includes: in response to content of the data capsule, performing a processing operation corresponding to a second event on behalf of at least one processing device, wherein the at least one processing device is associated with an application of a first type and a second application of a second type correspond.

The plurality of pools of an embodiment is coupled to a plurality of applications, the plurality of pools includes a plurality of data capsules corresponding to the plurality of applications, the plurality of pools provides access to the plurality of data capsules through the plurality of applications, wherein the plurality of At least two of the applications are different applications.

The multiple pools of an embodiment provide a stateful cache of multiple data capsules.

The multiple pools of an embodiment provide a linear ordering of multiple data capsules.

The data structures of an embodiment are untyped.

The data structure of the data capsule of an embodiment provides a platform-independent representation of event data and state information.

The data structure of the data capsule of an embodiment provides platform-independent access to event data and state information.

Transferring includes transferring a data capsule from a first application of a first application type to at least one second application of at least one second application type, wherein the first application type is different from the second application type , wherein generating the at least one data sequence is performed by a first application, the method comprising: keeping the at least one data sequence of the data capsule intact during transmission.

The method of an embodiment includes using at least one sequence of data during operation of the second application.

The method of an embodiment includes generating a first data set comprising event data for a source device of at least one processing device, the device event data including data specifying events registered by the source device, and identification data including data identifying the source device.

The method of an embodiment includes generating a second data set comprising the complete set of state information for the event, wherein each of the first data set and the second data set comprises a typed data bundle in a type-specific data layout.

The translating of an embodiment includes encapsulating the first data set and the second data set by forming a data capsule to include the first data set and the second data set, wherein the data capsule has an application-independent Represented data structure.

The method of an embodiment comprises: detecting an event of a first processing device running under an application program of a first type; generating a data sequence comprising event data of the first processing device, the event data specifying the event and state information of the event, wherein the event The data and state information are type-specific data having a type corresponding to the application; forming a data capsule to include a data sequence, the data capsule having a data structure including an application-independent representation of the data sequence; A second event of a second processing device running under at least one second application of two types, wherein the second type is different from the first type, and wherein the at least one processing device includes a first processing device and a second processing device; identifying a correspondence between the data capsule and the second event; and performing an operation in response to the second event using the contents of the data sequence of the data capsule.

Generating the data sequence of an embodiment includes generating a first data set comprising event data. Generating the data sequence of an embodiment includes generating a second data set including state information. Generating the data sequence of an embodiment includes forming the first data sequence to include the first data set and the second data set.

The event data of an embodiment is a tagged sequence of bytes representing typed data.

The event data of an embodiment includes a type header and a type-specific data layout.

The state information of an embodiment is a tagged sequence of bytes representing typed data.

The state information of an embodiment includes a type header and a type-specific data layout.

The method of an embodiment includes generating at least one offset. The method of an embodiment includes forming the data capsule to include at least one offset.

The method of an embodiment includes generating a first offset having a first variable length, wherein the first offset points to event data of a first data sequence of the at least one data sequence. The method of an embodiment includes: generating a second offset having a second variable length, wherein the second offset points to state information of a first data sequence in the at least one data sequence.

The method of an embodiment includes forming a first code path through the data capsule using a first offset of the at least one offset. The method of an embodiment includes forming a second code path through the data capsule using a second offset of the at least one offset, wherein the first code path and the second code path are different paths.

At least one of the first offset and the second offset of an embodiment includes metadata including context specific metadata corresponding to a context of the application.

The method of an embodiment includes transmitting the data capsule to a pool of the plurality of pools.

The method of an embodiment includes searching the plurality of pools for a data capsule corresponding to the second event. The method of an embodiment includes extracting a data capsule from the pool in response to identifying the correspondence.

The plurality of pools of an embodiment is coupled to the application and at least one second application, the plurality of pools includes a plurality of data capsules corresponding to the application and the at least one second application, the plurality of pools provides Access to multiple data capsules by a second application.

The multiple pools of an embodiment provide a stateful cache of multiple data capsules.

The multiple pools of an embodiment provide a linear ordering of multiple data capsules.

The data structures of an embodiment are untyped.

The data structure of the data capsule of an embodiment provides a platform-independent representation of event data and state information.

The data structure of the data capsule of an embodiment provides platform-independent access to event data and state information.

Embodiments described herein include a method comprising: dividing an application into a plurality of processes; using a process of the plurality of processes to generate a portion of the output of the application; encapsulating the portion of the output in a first data capsule, and transferring a first data capsule to at least one of a plurality of pools, wherein the plurality of pools includes a plurality of data capsules received from a plurality of processes; accessing the plurality of pools, and retrieving a second of the plurality of processes in the input a process, wherein the input is in a second data capsule of the plurality of data capsules; and coordinating processing among the plurality of processes using the plurality of data capsules and the plurality of pools.

Embodiments described herein include a system comprising: at least one processing device executing a plurality of processes including separable program execution contexts for a plurality of applications, wherein each application includes at least one processes; and a plurality of pools coupled to at least one processing device; the at least one processing device translates events of each of the plurality of processes into data capsules and transmits the data capsules to the plurality of pools, wherein the data capsules include initiated An application-independent representation of state information for a process and event data for an event of a data capsule; each process operates as a recognition process that recognizes in a plurality of pools including the interaction function corresponding to the recognition process and the identity of the recognition process A data capsule of at least one of the contents; the identification process retrieves the identified data capsule from the plurality of pools and performs processing appropriate to the content of the identified data capsule.

Embodiments described herein include a method comprising: detecting gestures performed by a body from gesture data, wherein the gesture data is received via a detector; and executing a plurality of processes on a processing device, the plurality of processes generating events, the events comprising representing gestures collection of events for each of multiple processes; translating events for each of multiple processes into data capsules; delivering data capsules to multiple pools; collection of processes in multiple processes operating as a recognition process that is in multiple pools identifying a data capsule comprising content corresponding to the gesture; and the identification process retrieves the identified data capsule from the plurality of pools and generates a gesture signal from the identified data capsule by synthesizing the contents of the identified data capsule to form the gesture signal, wherein, The attitude signal indicates attitude.

The plurality of processes of an embodiment includes a separable program execution context for a spatial manipulation application.

The gesture data of an embodiment is absolute three-spatial position data of the user's immediate state at a point in time and space.

The method of an embodiment includes recognizing gestures using only gesture data.

The detection of an embodiment includes: at least one of detecting the position of the body, detecting the direction of the body, and detecting the motion of the body.

The method of an embodiment includes recognizing a gesture, wherein recognizing includes recognizing a pose and an orientation of a body part.

The detecting of an embodiment includes detecting at least one of the first set of appendages and the second set of appendages of the body.

The detecting of an embodiment comprises dynamically detecting a position of at least one tag coupled to the body.

The detecting of an embodiment includes detecting a position of the set of tags coupled to the body.

Each tag in the set of tags of an embodiment includes a pattern, wherein each pattern of each tag in the set of tags is different from any pattern of any remaining tags of the plurality of tags.

The detection of an embodiment includes dynamically detecting and locating markers on the body.

The detecting of an embodiment comprises detecting a position of the set of markers coupled to the body.

The collection of markers of the embodiments form a plurality of patterns on the body.

The detecting of an embodiment includes detecting a position of the plurality of appendages of the body using a set of markers coupled to each of the plurality of appendages of the body.

The translating of an embodiment includes translating information of gestures into gesture symbols.

The gesture symbol of an embodiment represents a gesture vocabulary, and the gesture signal includes a communication of the gesture vocabulary.

The pose vocabulary of an embodiment represents the instant pose state of the kinematic linkages of the body in textual form.

The gesture vocabulary of an embodiment represents in textual form the orientation of the body's kinematic linkages.

The gesture vocabulary of an embodiment represents in textual form combinations of directions of kinematic linkages of the body.

The gesture vocabulary of an embodiment includes strings of characters representing states of kinematic linkages of the body.

The kinematic link of an embodiment is at least one first appendage of the body.

The method of an embodiment includes assigning each position in the string to a second appendage, the second appendage being connected to the first appendage.

The method of an embodiment includes assigning a character of the plurality of characters to each of the plurality of positions of the second appendage.

Various positions of an embodiment are established relative to a coordinate origin.

The method of an embodiment comprises: regardless of the general position and orientation of the body, and interactively in response to the movement of the body, using an absolute position and orientation in space, a fixed position and orientation relative to the body position to establish the coordinate origin.

The method of an embodiment includes assigning a character of the plurality of characters to each of the plurality of orientations of the first appendage.

The detecting of an embodiment comprises detecting when the extrapolated position of the body intersects a virtual space, wherein the virtual space comprises a space described on a display device coupled to the at least one processing device.

The method of an embodiment includes controlling a virtual object in a virtual space when an extrapolated position intersects the virtual object.

The controlling of an embodiment includes controlling the position of the virtual object in the virtual space in response to the extrapolated position in the virtual space.

The controlling of an embodiment includes controlling a gesture of the virtual object in the virtual space in response to the gesture.

The method of an embodiment includes controlling the ratio of detection and control to generate consistency between a virtual space comprising a space depicted on a display and a physical space comprising a space occupied by a body.

The method of an embodiment includes controlling at least one virtual object in the virtual space in response to movement of the at least one physical object in the physical space.

The method of an embodiment includes controlling a component using the gesture signal, the component being coupled to at least one processing device.

Control of components of an embodiment includes simultaneously controlling a three-dimensional object through six degrees of freedom by mapping poses to the three-dimensional object.

Control of components of an embodiment includes controlling a three-dimensional object through three translational degrees of freedom and three rotational degrees of freedom.

The three-spatial object of an embodiment is visualized on a display device coupled to at least one processing device.

The three-spatial object of an embodiment is a remote system coupled to a computer.

The method of an embodiment includes controlling motion of a three-space object by mapping poses to a plurality of object translations of the three-space object.

The mapping of an embodiment includes direct mapping between pose and translation of multiple objects.

The mapping of an embodiment includes indirect mapping between poses and translations of multiple objects.

The data capsule of an embodiment includes an application-independent representation of the state information of the process originating the data message and the event data of the event.

The method of an embodiment includes forming an interactive application from a plurality of processes using the data capsule and the plurality of pools by coordinating the operation of each of the plurality of processes.

The method of an embodiment includes coordinating operations of the plurality of processes using at least one of the data capsule and the plurality of pools.

The method of an embodiment includes: dividing an application program into a set of processes, wherein the plurality of processes includes a set of processes.

The method of an embodiment includes the process of generating an output by interactively processing a plurality of retrieved data capsules of at least one of the plurality of pools.

The multiple processes of an embodiment comprise separable program execution contexts of multiple application programs, wherein each application program comprises at least one process.

The method of an embodiment includes: executing multiple processes in parallel.

The method of an embodiment includes executing a first set of processes in parallel and executing a second set of processes sequentially, wherein the plurality of processes includes the first set of processes and the second set of processes.

An event of an embodiment represents process input.

An event of an embodiment represents process output.

The events of an embodiment include user interface events.

The events of an embodiment include graphics events.

An event of an embodiment represents a process state.

The process state of an embodiment represents the interactive function of the process, wherein the interactive function of the process is exposed to multiple processes as the content of the data capsule.

The method of an embodiment includes: defining an application programming interface (API) of a plurality of processes through the content of the data capsule, rather than defining the API through function calls.

The content of the data capsule of an embodiment is application-independent and recognizable by multiple processes.

The at least one processing device of an embodiment comprises a plurality of processing devices.

A first set of at least one process in the plurality of processes of the embodiment runs under the first set of at least one processing device in the plurality of processing devices, and a second set of at least one process in the plurality of processes runs on the plurality of processing devices The at least one processing device in the second set operates.

The plurality of processes of an embodiment includes a first process.

The translating of an embodiment includes transforming an event of the first process into at least one data sequence comprising first process event data specifying the event and state information of the event.

The first process event data and state information of an embodiment are type-specific data having a type corresponding to an application program of the first process.

The translating of an embodiment includes forming a data capsule to include at least one data sequence, the data capsule having a data structure that includes an application-independent representation of the at least one data sequence.

The plurality of processes of an embodiment includes a second process.

The translating of an embodiment includes transforming the state change event of the second process into at least one data sequence comprising second process event data specifying the event and state information of the event.

The second process event data and state information of an embodiment are type-specific data having a type corresponding to an application program of the second process.

The translating of an embodiment includes forming a data capsule to include at least one data sequence, the data capsule having a data structure that includes an application-independent representation of the at least one data sequence.

The identification process of an embodiment is a second process, and the retrieval includes retrieving the identified data capsules from the plurality of pools and performing the second process of processing appropriate to the contents of the identified data capsules.

The content of the identified data capsule of an embodiment is data representing state information of the first process.

The translating of an embodiment includes transforming the content of the identified data capsule into at least one new data sequence representing at least one of an event of the first process and an event of the second process.

The at least one new data sequence of an embodiment includes state information of at least one of the first process and the second process and event data specifying an event.

The event data and state information of at least one of the first process and the second process of an embodiment are type-specific data having a type corresponding to an application program of the at least one of the first process and the second process.

The translating of an embodiment includes forming a data capsule to include at least one new data sequence, the data capsule having a data structure that includes an application-independent representation of the at least one new data sequence.

Processes of an embodiment use at least one new data sequence.

Proper processing of the content of the identified data capsule of an embodiment includes rendering the graphical object, wherein the graphical object is rendered on a display of the at least one processing device.

The rendering of an embodiment includes direct rendering, wherein multiple processes draw directly to a graphics layer of at least one processing device, wherein multiple pools are used for coordination among the multiple processes as appropriate for rendering.

The rendering of an embodiment includes a plurality of processes that transmit data capsules including rendering commands to a plurality of pools. The rendering of an embodiment includes retrieving rendering commands from a plurality of pools, interpreting the rendering commands, and driving a plurality of processes of a graphics layer of at least one processing device in response to the rendering commands.

The rendering of an embodiment includes multiple processes rendering to pixel buffers. The rendering of an embodiment includes a plurality of processes that transfer raw frame data resulting from rendering to a pixel buffer to a plurality of pools. The rendering of an embodiment includes a plurality of processes retrieving raw frame data from a plurality of pools and combining the raw frame data for use in a graphics layer driving at least one processing device.

The method of an embodiment includes detecting events of a plurality of processes. The method of an embodiment comprises generating at least one data sequence comprising event data specifying an event and state information of the event, wherein the event data and state information are type-specific data having a type corresponding to an application of at least one processing device. The method of an embodiment includes forming a data capsule to include at least one data sequence, the data capsule having a data structure that includes an application-independent representation of the at least one data sequence.

The generating at least one data sequence of an embodiment comprises generating a first respective data set comprising first respective event data. The generating of at least one data sequence of an embodiment comprises generating a second respective data set comprising second respective state information. The generating of the at least one data sequence of an embodiment comprises forming the first data sequence to include a first respective data set and a second respective data set.

Generating the first respective data set of an embodiment comprises forming the first respective data set to include identification data of the at least one processing device, the identification data comprising data identifying the at least one processing device.

The generating at least one data sequence of an embodiment comprises generating a first respective data set comprising first respective event data. The generating of at least one data sequence of an embodiment comprises generating a second respective data set comprising second respective state information. The generating of the at least one data sequence of an embodiment comprises forming the second data sequence to include the first respective data set and the second respective data set.

The generating the first respective data set of an embodiment comprises: generating the first respective data set offset, wherein the first respective data set offset points to the first respective data set of the second data sequence.

The generating the second respective data set of an embodiment comprises: generating the second respective data set offset, wherein the second respective data set offset points to the second respective data set of the second data sequence.

The first respective data set of an embodiment is a description link list comprising a description of the data.

The event data of an embodiment is a tagged sequence of bytes representing typed data.

The event data of an embodiment includes a type header and a type-specific data layout.

The state information of an embodiment is a tagged sequence of bytes representing typed data.

The state information of an embodiment includes a type header and a type-specific data layout.

The method of an embodiment includes generating at least one offset. The method of an embodiment includes forming the data capsule to include at least one offset.

The method of an embodiment includes generating a first offset having a first variable length, wherein the first offset points to event data of a first data sequence of the at least one data sequence.

The method of an embodiment includes: generating a second offset having a second variable length, wherein the second offset points to state information of a first data sequence in the at least one data sequence.

The method of an embodiment includes forming a first code path through the data capsule using a first offset of the at least one offset. The method of an embodiment includes forming a second code path through the data capsule using a second offset of the at least one offset, wherein the first code path and the second code path are different paths.

At least one of the first offset and the second offset of an embodiment includes metadata including context specific metadata corresponding to a context of the application.

The method of an embodiment includes generating a header including a length of the data capsule. The method of an embodiment includes forming the data capsule to include the header.

The method of an embodiment includes transmitting the data capsule to a pool of the plurality of pools.

The method of an embodiment includes detecting a second event of at least one processing device. The method of an embodiment includes searching the plurality of pools for a data capsule corresponding to the second event.

The method of an embodiment includes identifying a correspondence between the data capsule and the second event. The method of an embodiment includes extracting a data capsule from the pool in response to identifying. The method of an embodiment includes: in response to content of the data capsule, performing a processing operation corresponding to a second event on behalf of at least one processing device, wherein the at least one processing device is associated with an application of a first type and a second application of a second type correspond.

The plurality of pools of an embodiment is coupled to a plurality of applications, the plurality of pools includes a plurality of data capsules corresponding to the plurality of applications, the plurality of pools provides access to the plurality of data capsules through the plurality of applications, wherein the plurality of At least two of the applications are different applications.

The multiple pools of an embodiment provide a stateful cache of multiple data capsules.

The multiple pools of an embodiment provide a linear ordering of multiple data capsules.

The data structures of an embodiment are untyped.

The data structure of the data capsule of an embodiment provides a platform-independent representation of event data and state information.

The data structure of the data capsule of an embodiment provides platform-independent access to event data and state information.

The transferring of an embodiment includes transferring the data capsule from a first application having a first application type to at least one second application having at least one second application type, wherein the first application type is the same as the second application Depending on the type of program, wherein generating the at least one data sequence is performed by a first application, the method includes keeping intact the at least one data sequence of the data capsule during the transfer.

The method of an embodiment includes using at least one sequence of data during operation of the second application.

The method of an embodiment includes generating a first data set comprising event data for a source device of at least one processing device, the device event data including data specifying events registered by the source device, and identification data including data identifying the source device.

The method of an embodiment includes generating a second data set comprising the complete set of state information for the event, wherein each of the first data set and the second data set comprises a typed data bundle in a type-specific data layout.

The translating of an embodiment includes encapsulating the first data set and the second data set by forming a data capsule to include the first data set and the second data set, wherein the data capsule has an application-independent Represented data structure.

The method of an embodiment includes detecting an event of a first processing device running under an application of a first type. The method of an embodiment includes generating a data sequence comprising event data of the first processing device, the event data specifying the event and state information for the event, wherein the event data and state information are type-specific data having a type corresponding to the application. The method of an embodiment includes forming a data capsule to include a data sequence, the data capsule having a data structure including an application-independent representation of the data sequence. The method of an embodiment comprises: detecting a second event of a second processing device running under at least one second application of at least one second type, wherein the second type is different from the first type, wherein the at least one processing device A first processing device and a second processing device are included. The method of an embodiment includes identifying a correspondence between the data capsule and the second event. The method of an embodiment comprises: using the contents of the data sequence of the data capsule, performing the operation in response to the second event.

Generating the data sequence of an embodiment includes generating a first data set comprising event data. Generating the data sequence of an embodiment includes generating a second data set including state information. Generating the data sequence of an embodiment includes forming the first data sequence to include the first data set and the second data set.

The event data of an embodiment is a tagged sequence of bytes representing typed data.

The event data of an embodiment includes a type header and a type-specific data layout.

The state information of an embodiment is a tagged sequence of bytes representing typed data.

The state information of an embodiment includes a type header and a type-specific data layout.

The method of an embodiment includes generating at least one offset. The method of an embodiment includes forming the data capsule to include at least one offset.

The method of an embodiment includes generating a first offset having a first variable length, wherein the first offset points to event data of a first data sequence of the at least one data sequence. The method of an embodiment includes: generating a second offset having a second variable length, wherein the second offset points to state information of a first data sequence in the at least one data sequence.

The method of an embodiment includes forming a first code path through the data capsule using a first offset of the at least one offset. The method of an embodiment includes forming a second code path through the data capsule using a second offset of the at least one offset, wherein the first code path and the second code path are different paths.

At least one of the first offset and the second offset of an embodiment includes metadata including context specific metadata corresponding to a context of the application.

The method of an embodiment includes transmitting the data capsule to a pool of the plurality of pools.

The method of an embodiment includes searching the plurality of pools for a data capsule corresponding to the second event. The method of an embodiment includes extracting a data capsule from the pool in response to identifying the correspondence.

The plurality of pools of an embodiment are coupled to the application and at least one second application, the plurality of pools include a plurality of data capsules corresponding to the application and the at least one second application, the plurality of pools provide Two applications have access to multiple data capsules.

The multiple pools of an embodiment provide a stateful cache of multiple data capsules.

The multiple pools of an embodiment provide a linear ordering of multiple data capsules.

The data structures of an embodiment are untyped.

The data structure of the data capsule of an embodiment provides a platform-independent representation of event data and state information.

The data structure of the data capsule of an embodiment provides access to pairs of event data and state information.

Embodiments described herein include a method comprising: executing a plurality of processes on a processing device, the plurality of processes including separable program execution contexts of a plurality of application programs, wherein each application program includes at least one process; events for each of the plurality of processes are translated into data messages, where the data messages include state information for the process originating the data message and an application-independent representation of the event data for the event; data messages are delivered to multiple pools at least one pool; coordinating among the processes, including: each process of the plurality of processes coordinating with a peer process in the plurality of processes by retrieving state information for the peer process from the plurality of pools; and by interactively A collection of data messages from at least one of the plurality of pools is combined to generate an output of the plurality of processes.

Embodiments described herein include a system comprising: a detector for receiving gesture data representing a gesture performed by a body; and a processor coupled to the detector, the processor automatically detects the gesture from the gesture data, the processor executes multiple processes, multiple processes generate events including a collection of events representing gestures, the processor translates the events of each of the multiple processes into data capsules, the processor transmits the data capsules to multiple pools, wherein multiple The collection of processes in the process operates as a recognition process, the recognition process identifies a data capsule comprising content corresponding to the pose in the plurality of pools, the recognition process retrieves the identified data capsule from the plurality of pools, and synthesizes the identified data capsule by The content is to form a gesture signal to generate a gesture signal from the recognized data capsule, wherein the gesture signal represents a gesture.

The systems and methods described herein include and/or operate under and/or in association with a processing system. As is known in the art, a processing system includes any collection of processor-based or computing devices, or components of a processing system or device, operating together. For example, a processing system may include one or more of a portable computer, a portable communication device operating in a communication network, and or a network server. A portable computer may be any number and/or combination of devices selected from among, but not limited to, a personal computer, a cellular telephone, a personal digital assistant, a portable computing device, and a portable communication device. A processing system may include components within a larger computer system.

The processing system of an embodiment includes at least one processor and at least one memory device or subsystem. The processing system may also include or be coupled to at least one database. The term "processor" is used generally herein to refer to any logical processing unit, such as one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), and the like. The processor and memory may be integrally integrated into a single chip, distributed among multiple chips or components in the host system, and/or provided by some combination of algorithms. The methods described herein may be implemented by one or more of software algorithms, programs, firmware, hardware, components, circuits in any combination.

System components implementing the systems and methods described herein may be located together or in separate locations. Accordingly, system components implementing the systems and methods described herein may be components of a single system, multiple systems, and/or geographically separated systems. These components may also be subcomponents or subsystems of a single system, multiple systems, and/or geographically separated systems. These components may be coupled to the host system or one or more other components of a system coupled to the host system.

Communication paths couple system components and include any medium used to communicate or transfer files among the components. Communication paths include wireless connections, wired connections, and hybrid wireless/wired connections. Communication paths also include couplings or connections to networks including local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), private networks, inter-office or back-end networks, and the Internet. In addition, communication paths include removable fixed media such as floppy disks, hard drives, and CD-ROM disks) and flash RAM, Universal Serial Bus (USB) connections, RS-232 connections, telephone lines, buses, and e-mail messages .

Unless the context clearly requires otherwise, throughout the specification the words "comprises", "comprising", etc. are to be read in an inclusive sense as opposed to an exclusive or exclusive sense, ie, "including, but not limited to". Words using the singular or the plural also include the plural or the singular, respectively. Additionally, the words "herein," "below," "above," "below," and words of similar import refer to this application as a whole and not to any particular portions of this application. When the word "or" is used with reference to a list of two or more items, that word covers all interpretations of any of the items in the list, all of the items in the list, and any combination of the items in the list.

The above description of embodiments is not intended to be exhaustive or to limit the described systems and methods to the precise forms disclosed. While specific embodiments, and examples, are described herein for illustrative purposes, various equivalent modifications are possible within the scope of other systems and methods, those skilled in the art will understand. The teachings provided herein are not only applicable to the systems and methods described above, but can also be applied to other processing systems and methods.

The elements and acts of the various above-described embodiments can be combined to provide other embodiments. These and other changes can be made to the embodiments in light of the above detailed description.

Generally, in the appended claims, the terms used should not be construed to limit the embodiments to the specific embodiments disclosed in the specification and claims, but rather should be understood to include all systems operating under the claims. Accordingly, the embodiments are not limited by the disclosure herein, but rather the scope of the embodiments is determined entirely by the claims.

Although certain aspects of the embodiments are presented in certain claim forms, the inventors contemplate the various aspects of the embodiments in any number of claim forms. Accordingly, the inventors reserve the right to add additional claims after filing the application to follow such additional claim forms for other aspects of the embodiments.

Claims (325)

1. a kind of multi-process exchange method, including：

Multiple processes are performed at least one processing equipment；

The event of each process in the multiple process is translated into data capsule；

The data capsule is sent to multiple ponds；

Each process is as identification process operation, and the identification process is recognized in the multiple pond to be included and the identification process Interactive function and the identification process the corresponding content of mark in the data capsule of at least one；And

The data capsule of identification process retrieval from the multiple pond, and perform the data capsule to the identification Content processing,

Wherein, data capsule includes initiating the state of a process information of data-message and the event data and application program of event Unrelated expression.

2. multi-process exchange method as claimed in claim 1, using the data capsule and the multiple pond by coordinating The operation for stating each process in multiple processes forms interactive application from the multiple process.

3. multi-process exchange method as claimed in claim 1, including：It is the multiple to coordinate using at least one in following The operation of process：The data capsule and the multiple pond.

4. multi-process exchange method as claimed in claim 1, including：Application program is divided into the set of process, wherein, institute Stating multiple processes includes the set of the process.

5. multi-process exchange method as claimed in claim 1, including by alternatively handling at least one in the multiple pond The data capsules of multiple retrievals in individual pond generates the process of output.

6. multi-process exchange method as claimed in claim 1, wherein, the multiple process includes dividing for multiple application programs From program perform context, wherein, each application program include at least one process.

7. multi-process exchange method as claimed in claim 1, including the multiple process is performed parallel.

8. multi-process exchange method as claimed in claim 1, including：The first set of parallel executive process, and sequentially hold The second set of traveling journey, wherein, the multiple process includes the first set of the process and the second set of the process.

9. multi-process exchange method as claimed in claim 1, wherein, the representations of events process input.

10. multi-process exchange method as claimed in claim 1, wherein, the representations of events process output.

11. multi-process exchange method as claimed in claim 1, wherein, the event includes user interface event.

12. multi-process exchange method as claimed in claim 1, wherein, the event includes graphical event.

13. multi-process exchange method as claimed in claim 1, wherein, the representations of events process status.

14. multi-process exchange method as claimed in claim 13, wherein, the process status represents the friendship of the process Mutual function, wherein, the interactive function of the process is as the content exposure of the data capsule in the multiple process.

15. multi-process exchange method as claimed in claim 14, including：Defined by the content of the data capsule described The application programming interface API of multiple processes, rather than define by funcall the API.

16. multi-process exchange method as claimed in claim 15, wherein, the content of the data capsule is unrelated with application program And can be by the multiple progress recognizing.

17. multi-process exchange method as claimed in claim 1, wherein, at least one described processing equipment includes multiple processing Equipment.

18. multi-process exchange method as claimed in claim 17, wherein, of at least one process in the multiple process One is integrated under the first set of at least one processing equipment in the multiple processing equipment in operation, the multiple process Run under the second set of at least one processing equipment of the second set of at least one process in the multiple processing equipment.

19. multi-process exchange method as claimed in claim 18, wherein, the multiple process includes the first process.

20. multi-process exchange method as claimed in claim 19, wherein, the translation includes：By the thing of first process Part is transformed to include at least one data of the status information for the first process event data and event for specifying the event Sequence.

21. multi-process exchange method as claimed in claim 20, wherein, the first process event data and status information are The specific data of type with type corresponding with the application program of first process.

22. multi-process exchange method as claimed in claim 21, wherein, the translation includes：The data capsule is formed It is to include at least one described data sequence, the data capsule has including at least one data sequence with applying journey The data structure of the unrelated expression of sequence.

23. multi-process exchange method as claimed in claim 19, wherein, the multiple process includes the second process.

24. multi-process exchange method as claimed in claim 23, wherein, the translation includes：By the shape of second process State change event be transformed to include the specified event the second process event data and the event status information at least One data sequence.

25. multi-process exchange method as claimed in claim 24, wherein, the second process event data and status information are The specific data of type with type corresponding with the application program of second process.

26. multi-process exchange method as claimed in claim 25, wherein, the translation includes：The data capsule is formed It is to include at least one described data sequence, the data capsule has including at least one data sequence with applying journey The data structure of the unrelated expression of sequence.

27. multi-process exchange method as claimed in claim 23, wherein, the identification process is second process, described Retrieval includes：The data capsule of retrieval from the multiple pond, and perform the content to the data capsule of the identification Processing the second process.

28. multi-process exchange method as claimed in claim 27, wherein, the content of the data capsule of the identification is to represent institute State the data of the first state of a process information.

29. multi-process exchange method as claimed in claim 28, wherein, the translation includes：By the data glue of the identification The content conversion of capsule is at least one new data sequence, and at least one described new data sequence represents first process At least one in the event of event and second process.

30. multi-process exchange method as claimed in claim 29, wherein, at least one described new data sequence includes described The event data of at least one state of a process information and the specified event in the first process and second process.

31. multi-process exchange method as claimed in claim 30, wherein, in first process and second process extremely The event data and status information of a few process be have with first process and second process at least The specific data of type of the corresponding type of application program of one process.

32. multi-process exchange method as claimed in claim 31, wherein, the translation includes：The data capsule is formed It is to include at least one described new data sequence, the data capsule, which has, includes at least one new data sequence The data structure of the expression unrelated with application program.

33. multi-process exchange method as claimed in claim 32, wherein, at least one is new using described for the multiple process Data sequence.

34. multi-process exchange method as claimed in claim 1, wherein, the content of the data capsule to the identification Processing includes rendering Drawing Object, wherein, the Drawing Object is rendered on the display of at least one processing equipment.

35. multi-process exchange method as claimed in claim 34, wherein, it is described to render including directly rendering, wherein described many Individual process is drawn directly into the graph layer of at least one processing equipment, wherein, the multiple pond is used to render for described Coordination between the multiple process.

36. multi-process exchange method as claimed in claim 34, wherein, it is described to render including following multiple processes：

The multiple pond will be sent to including the data capsule for rendering order；And

Render order from the retrieval of the multiple pond is described, explain described in render order, and render order in response to described and drive The graph layer of at least one processing equipment.

37. multi-process exchange method as claimed in claim 34, wherein, it is described to render including following multiple processes：

It is rendered into pixel buffer；

Original frame data is sent to the multiple pond, the original frame data as to being rendered described in the pixel buffer and Produce；And

Retrieve the original frame data from the multiple pond, and combine the original frame data, with described in driving at least Used in the graph layer of one processing equipment.

38. multi-process exchange method as claimed in claim 1, including：

Detect the event of the multiple process；

Generation includes specifying at least one data sequence of the status information of the event data of the event and the event, its In, the event data and status information are the classes with type corresponding with the application program of at least one processing equipment The specific data of type；And

Data capsule shape, which is turned into, includes at least one described data sequence, the data capsule have include it is described at least one The data structure of the expression unrelated with application program of data sequence.

39. multi-process exchange method as claimed in claim 38, wherein, at least one described data sequence of generation includes：

Generation includes the first respective data set of the first respective event data；

Generation includes the second respective data set of the second respective status information；And

First data sequence is formed as to include the described first respective data set and the second respective data set.

40. multi-process exchange method as claimed in claim 39, wherein, generating the first respective data set includes：By institute State the first respective data set to be formed as including the mark data of at least one processing equipment, the mark data includes mark The data of at least one processing equipment.

41. multi-process exchange method as claimed in claim 39, wherein, at least one described data sequence of generation includes：

Generation includes the first respective data set of the first respective event data；

Generation includes the second respective data set of the second respective status information；And

Second data sequence is formed as to include the described first respective data set and the second respective data set.

42. multi-process exchange method as claimed in claim 41, wherein, generating the first respective data set includes：Generation First respective data set offset, wherein, the first respective data set offset points to described the first of second data sequence Respective data set.

43. multi-process exchange method as claimed in claim 41, wherein, generating the second respective data set includes：Generation Second respective data set offset, wherein, the second respective data set offset points to described the second of second data sequence Respective data set.

44. multi-process exchange method as claimed in claim 39, wherein, the first respective data set is description chained list, institute Stating description chained list includes the description of the data.

45. multi-process exchange method as claimed in claim 38, wherein, the event data represents typed data The byte sequence of labeling.

46. multi-process exchange method as claimed in claim 45, wherein, the event data includes type header and type is special Fixed data layout.

47. multi-process exchange method as claimed in claim 38, wherein, the status information represents typed data The byte sequence of labeling.

48. multi-process exchange method as claimed in claim 47, wherein, the status information includes type header and type is special Fixed data layout.

49. multi-process exchange method as claimed in claim 38, including：

Generate at least one skew；And

The data capsule shape, which is turned into, includes at least one described skew.

50. multi-process exchange method as claimed in claim 49, including：

First skew of the generation with the first variable-length；

Wherein, the event data of the first data sequence at least one described data sequence is pointed in first skew.

51. multi-process exchange method as claimed in claim 49, including：

Second skew of the generation with the second variable-length；

Wherein, the status information of the first data sequence at least one described data sequence is pointed in second skew.

52. multi-process exchange method as claimed in claim 49, including：

Pass through data capsule formation first code path using the first skew at least one described skew；

Pass through data capsule formation second code path using the second skew at least one described skew；

Wherein, the first code path and the second code path are different paths.

53. multi-process exchange method as claimed in claim 52, wherein, in first skew and the described second skew extremely Few one includes metadata, and the metadata includes context certain metadata corresponding with the context of the application program.

54. multi-process exchange method as claimed in claim 38, including：

Generation includes the head of the length of the data capsule；

The data capsule shape, which is turned into, includes the head.

55. multi-process exchange method as claimed in claim 38, including：The data capsule is sent in the multiple pond Pond.

56. multi-process exchange method as claimed in claim 55, including：

The second event of detection at least one processing equipment；

Data capsule corresponding with the second event is searched in the multiple pond.

57. multi-process exchange method as claimed in claim 56, including：

Recognize the correspondence between the data capsule and the second event；

The data capsule is extracted from the pond in response to the identification；And

In response to the content of the data capsule, at least one processing equipment execution is represented corresponding with the second event Processing operation, wherein, the application program of described at least one processing equipment and the first kind and the second application of Second Type Program correspondence.

58. multi-process exchange method as claimed in claim 55, wherein, multiple application programs, institute are coupled in the multiple pond Stating multiple ponds includes multiple data capsules corresponding with the multiple application program, and the multiple pond is provided is answered by the multiple Access with program to the multiple data capsule, wherein, at least two application programs in the multiple application program are not Same application program.

59. multi-process exchange method as claimed in claim 55, wherein, the multiple pond provides the state of multiple data capsules Caching.

60. multi-process exchange method as claimed in claim 55, wherein, the multiple pond provides the linear of multiple data capsules Sequence.

61. multi-process exchange method as claimed in claim 38, wherein, the data structure is non-typed.

62. multi-process exchange method as claimed in claim 38, wherein, the data structure of the data capsule provides the thing Number of packages evidence and the status information are represented with platform-independent.

63. multi-process exchange method as claimed in claim 38, wherein, the data structure of the data capsule is provided to described Event data and the status information are accessed with platform-independent.

64. multi-process exchange method as claimed in claim 38, wherein, the transmission includes：By the data capsule from tool The first application program for having the first Application Type is sent at least one with least one the second Application Type Second application program, wherein, first Application Type is different from second Application Type, wherein, generate institute At least one data sequence is stated by first application program to perform, methods described includes：Kept during the transmission At least one data sequence of the data capsule is intact.

65. the multi-process exchange method as described in claim 64, including used during the operation of second application program At least one described data sequence.

66. multi-process exchange method as claimed in claim 38, including：Generation includes the source of at least one processing equipment The event data of equipment and the first data set of mark data, the device events data include specifying the source device registration The data of event, the mark data includes the data for identifying the source device.

67. the multi-process exchange method as described in claim 66, including：Generation includes the complete of the status information of the event Second data set of set, wherein, each in first data set and second data set is specific including type Typed data bundle in data layout.

68. the multi-process exchange method as described in claim 67, wherein, the translation includes：By the way that data capsule is formed It is to include first data set and second data set to encapsulate first data set and second data set, its In, the data capsule has the data structure for the expression unrelated with application program for including at least one data sequence.

69. multi-process exchange method as claimed in claim 38, including：

Detect the event for the first processing equipment run under the application program of the first kind；

Generation includes the data sequence of the event data of first processing equipment, and the event data specifies the event and institute The status information of event is stated, wherein, the event data and status information are with type corresponding with the application program The specific data of type；

Data capsule shape, which is turned into, includes the data sequence, and the data capsule is with including the data sequence and application The data structure of the unrelated expression of program；

Detect the of the second processing equipment run under with the application program of at least one of at least one Second Type second Two events, wherein, the Second Type is different from the first kind, wherein, at least one described processing equipment includes described First processing equipment and the second processing equipment；

Recognize the correspondence between the data capsule and the second event；And

Operation is performed using the content of the data sequence of the data capsule, in response to the second event.

70. the multi-process exchange method as described in claim 69, wherein, generating the data sequence includes：

Generation includes the first data set of the event data；

Generation includes the second data set of the status information；And

First data sequence is formed as to include first data set and second data set.

71. the multi-process exchange method as described in claim 69, wherein, the event data represents typed data The byte sequence of labeling.

72. the multi-process exchange method as described in claim 71, wherein, the event data includes type header and type is special Fixed data layout.

73. the multi-process exchange method as described in claim 69, wherein, the status information represents typed data The byte sequence of labeling.

74. the multi-process exchange method as described in claim 73, wherein, the status information includes type header and type is special Fixed data layout.

75. the multi-process exchange method as described in claim 69, including：

Generate at least one skew；And

The data capsule shape, which is turned into, includes at least one described skew.

76. the multi-process exchange method as described in claim 75, including：

First skew of the generation with the first variable-length, wherein, at least one described data sequence is pointed in first skew In the first data sequence event data；And

Second skew of the generation with the second variable-length, wherein, at least one described data sequence is pointed in second skew In the first data sequence status information.

77. the multi-process exchange method as described in claim 75, including：

Pass through data capsule formation first code path using the first skew at least one described skew；

Pass through data capsule formation second code path using the second skew at least one described skew；

Wherein, the first code path and the second code path are different paths.

78. the multi-process exchange method as described in claim 76, wherein, in first skew and the described second skew extremely Few one includes metadata, and the metadata includes context certain metadata corresponding with the context of the application program.

79. the multi-process exchange method as described in claim 69, including the data capsule is sent in the multiple pond Pond.

80. the multi-process exchange method as described in claim 79, including：

Data capsule corresponding with the second event is searched in the multiple pond；And

The data capsule is extracted in identification in response to the correspondence from the pond.

81. the multi-process exchange method as described in claim 79, wherein, the application program and institute are coupled in the multiple pond At least one second application program is stated, the multiple pond includes and the application program and at least one described second application program Corresponding multiple data capsules, the multiple pond, which is provided, passes through the application program and at least one described second application program pair The access of the multiple data capsule.

82. the multi-process exchange method as described in claim 79, wherein, the multiple pond provides the state of multiple data capsules Caching.

83. the multi-process exchange method as described in claim 79, wherein, the multiple pond provides the linear of multiple data capsules Sequence.

84. the multi-process exchange method as described in claim 69, wherein, the data structure is non-typed.

85. the multi-process exchange method as described in claim 69, wherein, the data structure of the data capsule provides the thing Number of packages evidence and the status information are represented with platform-independent.

86. the multi-process exchange method as described in claim 69, wherein, the data structure of the data capsule is provided to described Event data and the status information are accessed with platform-independent.

87. a kind of multi-process interactive systems, including：

At least one processing equipment, the processing equipment performs multiple processes；And

Multiple ponds, are coupled at least one described processing equipment；

The event of each process in the multiple process is translated to data capsule by least one described processing equipment, and will The data capsule is sent to the multiple pond；

Each process in the multiple process is as identification process operation, and the identification process recognizes bag in the multiple pond Include and the data glue of at least one in the interactive function of the identification process and the corresponding content of mark of the identification process Capsule；

The data capsule of identification process retrieval from the multiple pond, and perform the data capsule to the identification Content processing,

Wherein, data capsule include initiating the event data of the state of a process information of the data capsule and event with application The unrelated expression of program.

88. a kind of multi-process exchange method, including：

Multiple processes are performed at least one processing equipment, the multiple process includes the separable journey of multiple application programs Sequence performs context, wherein, each application program includes at least one process；

The event of each process in the multiple process is translated into data capsule, wherein, data capsule includes initiating described The expression unrelated with application program of the state of a process information of data capsule and the event data of event；

The data capsule is sent to multiple ponds；

Each process is as identification process operation, and the identification process is recognized in the multiple pond to be included and the identification process Interactive function and the identification process the corresponding content of mark in the data capsule of at least one；And

The data capsule of identification process retrieval from the multiple pond, and perform the data capsule to the identification Content processing.

89. the multi-process exchange method as described in claim 88, using the data capsule and the multiple pond by coordinating The operation of each process in the multiple process forms interactive application from the multiple process.

90. the multi-process exchange method as described in claim 88, including：It is described many to coordinate using at least one in following The operation of individual process：The data capsule and the multiple pond.

91. the multi-process exchange method as described in claim 88, including：Application program is divided into the set of process, wherein, The multiple process includes the set of the process.

92. the multi-process exchange method as described in claim 88, including process is by alternatively handling in the multiple pond The data capsules of multiple retrievals at least one pond generates output.

93. the multi-process exchange method as described in claim 88, including：The multiple process is performed parallel.

94. the multi-process exchange method as described in claim 88, including：The first set of parallel executive process, and order The second set of executive process, wherein, the multiple process includes the first set of the process and the second collection of the process Close.

95. the multi-process exchange method as described in claim 88, wherein, the representations of events process input.

96. the multi-process exchange method as described in claim 88, wherein, the representations of events process output.

97. the multi-process exchange method as described in claim 88, wherein, the event includes user interface event.

98. the multi-process exchange method as described in claim 88, wherein, the event includes graphical event.

99. the multi-process exchange method as described in claim 88, wherein, the representations of events process status.

100. the multi-process exchange method as described in claim 99, wherein, the process status represents the described of the process Interactive function, wherein, the interactive function of the process is as the content exposure of the data capsule in the multiple process.

101. the multi-process exchange method as described in claim 100, including：By the content of the data capsule to define The application programming interface API of multiple processes is stated, rather than defines by funcall the API.

102. the multi-process exchange method as described in claim 101, wherein, the content of the data capsule is and application program It is unrelated and can be by the multiple progress recognizing.

103. the multi-process exchange method as described in claim 88, wherein, at least one described processing equipment includes multiple places Manage equipment.

104. the multi-process exchange method as described in claim 103, wherein, at least one process in the multiple process Run under the first set of at least one processing equipment of the first set in the multiple processing equipment, in the multiple process At least one process at least one processing equipment of the second set in the multiple processing equipment second set under transport OK.

105. the multi-process exchange method as described in claim 88, wherein, the multiple process includes the first process.

106. the multi-process exchange method as described in claim 105, wherein, the translation includes：By first process Event is transformed to include at least one number of the status information for the first process event data and event for specifying the event According to sequence.

107. the multi-process exchange method as described in claim 106, wherein, the first process event data and status information It is the specific data of type with type corresponding with the application program of first process.

108. the multi-process exchange method as described in claim 107, wherein, the translation includes：By the data capsule shape As including at least one described data sequence, the data capsule has including at least one data sequence and application The data structure of the unrelated expression of program.

109. the multi-process exchange method as described in claim 105, wherein, the multiple process includes the second process.

110. the multi-process exchange method as described in claim 109, wherein, the translation includes：By second process State change event be transformed to include to specify the event the second process event data and the event status information extremely A few data sequence.

111. the multi-process exchange method as described in claim 110, wherein, the second process event data and status information It is the specific data of type with type corresponding with the application program of second process.

112. the multi-process exchange method as described in claim 111, wherein, the translation includes：By the data capsule shape As including at least one described data sequence, the data capsule has including at least one data sequence and application The data structure of the unrelated expression of program.

113. the multi-process exchange method as described in claim 109, wherein, the identification process is second process, institute Stating retrieval includes the data capsule of the retrieval from the multiple pond, and performs the content to the data capsule of the identification Processing the second process.

114. the multi-process exchange method as described in claim 113, wherein, the content of the data capsule of the identification is to represent The data of the first state of a process information.

115. the multi-process exchange method as described in claim 114, wherein, the translation includes：By the data of the identification The content conversion of capsule is at least one new data sequence, and at least one described new data sequence represents first process Event and second process event at least one.

116. the multi-process exchange method as described in claim 115, wherein, at least one described new data sequence includes institute State at least one state of a process information in the first process and second process and specify the event data of the event.

117. the multi-process exchange method as described in claim 116, wherein, in first process and second process The event data and status information of at least one process be have with first process and second process extremely The specific data of type of the corresponding type of application program of a few process.

118. the multi-process exchange method as described in claim 117, wherein, the translation includes：By the data capsule shape As including at least one described new data sequence, the data capsule, which has, includes at least one described new data sequence The expression unrelated with application program data structure.

119. the multi-process exchange method as described in claim 118, wherein, at least one is new described in the multiple process use Data sequence.

120. the multi-process exchange method as described in claim 88, wherein, the multiple process includes input process, described defeated Enter process and receive incoming event from input equipment.

121. the multi-process exchange method as described in claim 120, wherein, the translation includes：By the input equipment The incoming event be transformed to include to specify the event input equipment event data and the event status information extremely A few data sequence.

122. the multi-process exchange method as described in claim 121, wherein, the input equipment event data and status information It is the specific data of type with type corresponding with the application program of source device.

123. the multi-process exchange method as described in claim 122, wherein, the translation includes：By the data capsule shape As including at least one described data sequence, the data capsule has including at least one data sequence and application The data structure of the unrelated expression of program.

124. the multi-process exchange method as described in claim 88, wherein, the multiple process includes pointer process.

125. the multi-process exchange method as described in claim 124, wherein, the identification process is the pointer process, institute Stating retrieval includes：The data capsule of the identification is retrieved from the multiple pond, and performs the data capsule to the identification Content processing pointer process.

126. the multi-process exchange method as described in claim 125, wherein, the content of the data capsule of the identification is to represent The data of incoming event from input process.

127. the multi-process exchange method as described in claim 125, wherein, the content of the data capsule of the identification is to represent The data of the position of user's positive order pointer object of at least one processing equipment on display.

128. the multi-process exchange method as described in claim 127, wherein, the translation includes：By the data of the identification The content conversion of capsule be at least one new data sequence, at least one described new data sequence definition pointer object on The position of display.

129. the multi-process exchange method as described in claim 128, wherein, at least one described new data sequence includes referring to The pointer process event data of the fixed event and the status information of the pointer process event.

130. the multi-process exchange method as described in claim 129, wherein, the pointer process event data and status information It is the specific data of type with type corresponding with the application program of the pointer process.

131. the multi-process exchange method as described in claim 130, wherein, the translation includes：By the data capsule shape As including at least one described new data sequence, the data capsule, which has, includes at least one described new data sequence The expression unrelated with application program data structure.

132. the multi-process exchange method as described in claim 131, wherein, at least one is new described in the multiple process use Data sequence render the pointer object on the display.

133. the multi-process exchange method as described in claim 88, wherein, the multiple process includes graphic processes.

134. the multi-process exchange method as described in claim 133, wherein, the translation includes：By the graphic processes State change event be transformed to include to specify the event graphic processes event data and the event status information extremely A few data sequence.

135. the multi-process exchange method as described in claim 134, wherein, the graphic processes event data and status information It is the specific data of type with type corresponding with the application program of the graphic processes.

136. the multi-process exchange method as described in claim 135, wherein, the translation includes：By the data capsule shape As including at least one described data sequence, the data capsule has including at least one data sequence and application The data structure of the unrelated expression of program.

137. the multi-process exchange method as described in claim 133, wherein, the identification process is the graphic processes, institute Stating retrieval includes：The data capsule of retrieval from the multiple pond, and perform in the data capsule of the identification The graphic processes of the processing of appearance.

138. the multi-process exchange method as described in claim 137, wherein, the content of the data capsule of the identification is to represent The data of another state of a process information in the multiple process.

139. the multi-process exchange method as described in claim 138, wherein, the status information includes spatiality and pattern The information of at least one in state.

140. the multi-process exchange method as described in claim 137, wherein, the content of the data capsule of the identification is to represent The data of the position of user's positive order pointer object of at least one processing equipment on display.

The 141. multi-process exchange method as described in claim 140, wherein, the position of the pointer object is in Drawing Object In border, wherein, the Drawing Object is rendered by the graphic processes.

The 142. multi-process exchange method as described in claim 141, wherein, the translation includes：By the data of the identification The content conversion of capsule is at least one new data sequence, and at least one described new data sequence represents the figure pair As, the pointer object and the pointer object and at least one in the overlapping part on the border.

The 143. multi-process exchange method as described in claim 142, wherein, at least one described new data sequence includes referring to The graphic processes event data of the fixed event and the status information of the graphic processes event.

The 144. multi-process exchange method as described in claim 143, wherein, the graphic processes event data and status information It is the specific data of type with type corresponding with the application program of the graphic processes.

The 145. multi-process exchange method as described in claim 144, wherein, the translation includes：By the data capsule shape As including at least one described new data sequence, the data capsule, which has, includes at least one described new data sequence The expression unrelated with application program data structure.

The 146. multi-process exchange method as described in claim 145, wherein, at least one is new described in the multiple process use Data sequence render at least one in the Drawing Object and the pointer object on the display.

The 147. multi-process exchange method as described in claim 88, wherein, the content of the data capsule to the identification Processing include render Drawing Object, wherein, the Drawing Object is rendered on the display of at least one processing equipment.

The 148. multi-process exchange method as described in claim 147, wherein, it is described to render including directly rendering, wherein described Multiple processes are drawn directly into the graph layer of at least one processing equipment, wherein, the multiple pond is used for for the wash with watercolours Contaminate the coordination between the multiple process.

The 149. multi-process exchange method as described in claim 147, wherein, it is described to render including following multiple processes：

The multiple pond will be sent to including the data capsule for rendering order；And

Render order from the retrieval of the multiple pond is described, explain described in render order, and render order in response to described and drive The graph layer of at least one processing equipment.

The 150. multi-process exchange method as described in claim 147, wherein, it is described to render including following multiple processes：

It is rendered into pixel buffer；

Original frame data is sent to the multiple pond, the original frame data as to being rendered described in the pixel buffer and Produce；And

Retrieve the original frame data from the multiple pond, and combine the original frame data, with described in driving at least Used in the graph layer of one processing equipment.

The 151. multi-process exchange method as described in claim 88, including：

Detect the event of the multiple process；

Generation includes specifying at least one data sequence of the status information of the event data of the event and the event, its In, the event data and status information are the classes with type corresponding with the application program of at least one processing equipment The specific data of type；And

Data capsule shape, which is turned into, includes at least one described data sequence, the data capsule have include it is described at least one The data structure of the expression unrelated with application program of data sequence.

The 152. multi-process exchange method as described in claim 151, wherein, the bag of generation at least one data sequence Include：

Generation includes the first respective data set of the first respective event data；

Generation includes the second respective data set of the second respective status information；And

First data sequence is formed as to include the described first respective data set and the second respective data set.

The 153. multi-process exchange method as described in claim 152, wherein, generating the first respective data set includes：Will The first respective data set is formed as including the mark data of at least one processing equipment, and the mark data includes mark Know the data of at least one processing equipment.

The 154. multi-process exchange method as described in claim 152, wherein, at least one described data sequence of generation includes：

Generation includes the first respective data set of the first respective event data；

Generation includes the second respective data set of the second respective status information；And

Second data sequence is formed as to include the described first respective data set and the second respective data set.

The 155. multi-process exchange method as described in claim 154, wherein, generating the first respective data set includes：It is raw Into the first respective data set offset, wherein, the first respective data set offset points to described the of second data sequence One respective data set.

The 156. multi-process exchange method as described in claim 154, wherein, generating the second respective data set includes：It is raw Into the second respective data set offset, wherein, the second respective data set offset points to described the of second data sequence Two respective data sets.

The 157. multi-process exchange method as described in claim 152, wherein, the first respective data set is description chained list, The description chained list includes the description of the data.

The 158. multi-process exchange method as described in claim 151, wherein, the event data is to represent typed data Labeling byte sequence.

The 159. multi-process exchange method as described in claim 158, wherein, the event data includes type header and type Specific data layout.

The 160. multi-process exchange method as described in claim 151, wherein, the status information is to represent typed data Labeling byte sequence.

The 161. multi-process exchange method as described in claim 160, wherein, the status information includes type header and type Specific data layout.

The 162. multi-process exchange method as described in claim 151, including：

Generate at least one skew；And

The data capsule shape, which is turned into, includes at least one described skew.

The 163. multi-process exchange method as described in claim 162, including：

First skew of the generation with the first variable-length；

Wherein, the event data of the first data sequence at least one described data sequence is pointed in first skew.

The 164. multi-process exchange method as described in claim 162, including：

Second skew of the generation with the second variable-length；

Wherein, the status information of the first data sequence at least one described data sequence is pointed in second skew.

The 165. multi-process exchange method as described in claim 162, including：

Pass through data capsule formation first code path using the first skew at least one described skew；

Pass through data capsule formation second code path using the second skew at least one described skew；

Wherein, the first code path and the second code path are different paths.

The 166. multi-process exchange method as described in claim 165, wherein, in first skew and the described second skew At least one includes metadata, and the metadata includes the specific first number of context corresponding with the context of the application program According to.

The 167. multi-process exchange method as described in claim 151, including：

Generation includes the head of the length of the data capsule；

The data capsule shape, which is turned into, includes the head.

The 168. multi-process exchange method as described in claim 151, including：The data capsule is sent to the multiple pond In pond.

The 169. multi-process exchange method as described in claim 168, including：

The second event of detection at least one processing equipment；

Data capsule corresponding with the second event is searched in the multiple pond.

The 170. multi-process exchange method as described in claim 169, including：

Recognize the correspondence between the data capsule and the second event；

The data capsule is extracted from the pond in response to the identification；And

In response to the content of the data capsule, at least one processing equipment execution is represented corresponding with the second event Processing operation, wherein, the application program of described at least one processing equipment and the first kind and the second application of Second Type Program correspondence.

The 171. multi-process exchange method as described in claim 168, wherein, multiple application programs are coupled in the multiple pond, The multiple pond includes multiple data capsules corresponding with the multiple application program, and the multiple pond is provided by the multiple Access of the application program to the multiple data capsule, wherein, at least two application programs in the multiple application program are Different application programs.

The 172. multi-process exchange method as described in claim 168, wherein, the multiple pond provides the shape of multiple data capsules State is cached.

The 173. multi-process exchange method as described in claim 168, wherein, the multiple pond provides the line of multiple data capsules Property sequence.

The 174. multi-process exchange method as described in claim 151, wherein, the data structure is non-typed.

The 175. multi-process exchange method as described in claim 151, wherein, the data structure of the data capsule provides described Event data and the status information are represented with platform-independent.

The 176. multi-process exchange method as described in claim 151, wherein, the data structure of the data capsule is provided to institute State being accessed with platform-independent for event data and the status information.

The 177. multi-process exchange method as described in claim 151, wherein, the transmission includes：By the data capsule from The first application program with the first Application Type is sent at least one with least one the second Application Type Individual second application program, wherein, first Application Type is different from second Application Type, wherein, generation At least one described data sequence is performed by first application program, and methods described includes：Protected during the transmission At least one data sequence for holding the data capsule is intact.

The 178. multi-process exchange method as described in claim 177, including make during the operation of second application program With at least one described data sequence.

The 179. multi-process exchange method as described in claim 151, including：Generation includes at least one processing equipment The event data of source device and the first data set of mark data, the device events data include specifying the source device registration Event data, the mark data includes the data for identifying the source device.

The 180. multi-process exchange method as described in claim 179, including：Generation includes the complete of the status information of the event Second data set of universal class, wherein, each in first data set and second data set is specific including type Data layout in typed data bundle.

The 181. multi-process exchange method as described in claim 180, wherein, the translation includes：By by data capsule shape As encapsulating first data set and second data set including first data set and second data set, its In, the data capsule has the data structure for the expression unrelated with application program for including at least one data sequence.

The 182. multi-process exchange method as described in claim 151, including：

Detect the event for the first processing equipment run under the application program of the first kind；

Generation includes the data sequence of the event data of first processing equipment, and the event data specifies the event and institute The status information of event is stated, wherein, the event data and status information are the types with type corresponding with application program Specific data；

Data capsule shape, which is turned into, includes the data sequence, and the data capsule is with including the data sequence and application The data structure of the unrelated expression of program；

Detect the of the second processing equipment run under with the application program of at least one of at least one Second Type second Two events, wherein, the Second Type is different from the first kind, wherein, at least one described processing equipment includes described First processing equipment and the second processing equipment；

Recognize the correspondence between the data capsule and the second event；And

Operation is performed using the content of the data sequence of the data capsule, in response to the second event.

The 183. multi-process exchange method as described in claim 182, wherein, generating the data sequence includes：

Generation includes the first data set of the event data；

Generation includes the second data set of the status information；And

First data sequence is formed as to include first data set and second data set.

The 184. multi-process exchange method as described in claim 182, wherein, the event data is to represent typed data Labeling byte sequence.

The 185. multi-process exchange method as described in claim 184, wherein, the event data includes type header and type Specific data layout.

The 186. multi-process exchange method as described in claim 182, wherein, the status information is to represent typed data Labeling byte sequence.

The 187. multi-process exchange method as described in claim 186, wherein, the status information includes type header and type Specific data layout.

The 188. multi-process exchange method as described in claim 182, including：

Generate at least one skew；And

The data capsule shape, which is turned into, includes at least one described skew.

The 189. multi-process exchange method as described in claim 188, including：

First skew of the generation with the first variable-length, wherein, at least one described data sequence is pointed in first skew In the first data sequence event data；And

Second skew of the generation with the second variable-length, wherein, at least one described data sequence is pointed in second skew In the first data sequence status information.

The 190. multi-process exchange method as described in claim 188, including：

Pass through data capsule formation first code path using the first skew at least one described skew；

Pass through data capsule formation second code path using the second skew at least one described skew；

Wherein, the first code path and the second code path are different paths.

The 191. multi-process exchange method as described in claim 189, wherein, in first skew and the described second skew At least one includes metadata, and the metadata includes the specific first number of context corresponding with the context of the application program According to.

The 192. multi-process exchange method as described in claim 182, including：The data capsule is sent to the multiple pond In pond.

The 193. multi-process exchange method as described in claim 192, including：

Data capsule corresponding with the second event is searched in the multiple pond；And

The data capsule is extracted in identification in response to the correspondence from the pond.

The 194. multi-process exchange method as described in claim 192, wherein, the multiple pond be coupled to the application program and At least one described second application program, the multiple pond include with the application program and it is described at least one second apply journey The corresponding multiple data capsules of sequence, the multiple pond, which is provided, passes through the application program and at least one described second application program Access to the multiple data capsule.

The 195. multi-process exchange method as described in claim 192, wherein, the multiple pond provides the shape of multiple data capsules State is cached.

The 196. multi-process exchange method as described in claim 192, wherein, the multiple pond provides the line of multiple data capsules Property sequence.

The 197. multi-process exchange method as described in claim 182, wherein, the data structure is non-typed.

The 198. multi-process exchange method as described in claim 182, wherein, the data structure of the data capsule provides described Event data and the status information are represented with platform-independent.

The 199. multi-process exchange method as described in claim 182, wherein, the data structure of the data capsule is provided to institute State being accessed with platform-independent for event data and the status information.

A kind of 200. multi-process interactive systems, including：

At least one processing equipment, the processing equipment performs multiple processes, and the multiple process includes multiple application programs Separable program performs context, wherein, each application program includes at least one process；And

Multiple ponds, are coupled at least one described processing equipment；

The event of each process in the multiple process is translated to data capsule by least one described processing equipment, and will The data capsule is sent to multiple ponds, wherein, data capsule include initiating the state of a process information of the data capsule and The expression unrelated with application program of the event data of event；

Each process is as identification process operation, and the identification process is recognized in the multiple pond to be included and the identification process Interactive function and the identification process the corresponding content of mark in the data capsule of at least one；

The data capsule of identification process retrieval from the multiple pond, and perform the data capsule to the identification Content processing.

A kind of 201. multi-process exchange methods, including：

The posture that body is carried out is detected from attitude data, wherein, receive the attitude data via detector；

Multiple processes are performed on a processing device, and the multiple process generates event, and the event includes representing the posture The set of event；

The event of each process in the multiple process is translated into data capsule；

The data capsule is sent to multiple ponds；

The set of process in the multiple process is recognized as identification process operation, the identification process in the multiple pond Include the data capsule of content corresponding with the posture；And

The data capsule of identification process retrieval from the multiple pond, and by synthesizing the data glue of the identification The content of capsule to be formed the attitude signal to generate attitude signal from the data capsule of the identification, wherein, the posture letter Number the posture is represented,

Wherein, data capsule include initiating the event data of the state of a process information of the data capsule and event with application The unrelated expression of program.

The 202. multi-process exchange method as described in claim 201, wherein, the multiple process includes spatial operation application journey The separable program of sequence performs context.

The 203. multi-process exchange method as described in claim 201, wherein, the attitude data is user in time and space In point at immediate status absolute three spatial position data.

The 204. multi-process exchange method as described in claim 201, including recognize using only the attitude data appearance State.

The 205. multi-process exchange method as described in claim 201, wherein, the detection includes：Detect the position of the body Put, detect at least one in the direction of the body and the motion of the detection body.

The 206. multi-process exchange method as described in claim 201, including：The posture is recognized, wherein, the identification includes Recognize posture and the direction of the part of the body.

The 207. multi-process exchange method as described in claim 201, wherein, the detection includes：Detect the attached of the body At least one in the first set of limb and the second set of appendage.

The 208. multi-process exchange method as described in claim 201, wherein, the detection includes dynamically detection and is coupled to institute State the position of at least one label of body.

The 209. multi-process exchange method as described in claim 208, wherein, the detection includes detection and is coupled to the body Label set position.

The 210. multi-process exchange method as described in claim 209, wherein, each label in the set of the label includes Pattern, wherein each pattern of each label in the set of the label and remaining any label in the multiple label Any pattern is different.

The 211. multi-process exchange method as described in claim 201, wherein, the detection includes dynamically detecting and positioning Mark on the body.

The 212. multi-process exchange method as described in claim 211, wherein, the detection includes detection and is coupled to the body Mark set position.

The 213. multi-process exchange method as described in claim 211, wherein, being integrated on the body for the mark is formed Multiple patterns.

The 214. multi-process exchange method as described in claim 211, wherein, the detection includes：Using being coupled to the body The position of the multiple appendages for gathering to detect the body of the mark of each in multiple appendages of body.

The 215. multi-process exchange method as described in claim 201, wherein, the translation includes turning the information of the posture It is translated into posture symbol.

The 216. multi-process exchange method as described in claim 215, wherein, the posture symbol represents posture vocabulary, and The attitude signal includes the communication of the posture vocabulary.

The 217. multi-process exchange method as described in claim 216, wherein, the posture vocabulary represents described in the form of text The instant posture state of the kinematics link of body.

The 218. multi-process exchange method as described in claim 216, wherein, the posture vocabulary represents described in the form of text The direction of the kinematics link of body.

The 219. multi-process exchange method as described in claim 216, wherein, the posture vocabulary represents described in the form of text The combination in the direction of the kinematics link of body.

The 220. multi-process exchange method as described in claim 216, wherein, the posture vocabulary includes representing the body The string of the character of the state of kinematics link.

The 221. multi-process exchange method as described in claim 220, wherein, kinematics link be the body at least One the first appendage.

The 222. multi-process exchange method as described in claim 221, including each position in the string is distributed to second Appendage, second appendage is connected to first appendage.

The 223. multi-process exchange method as described in claim 222, including the character in multiple characters is distributed to described Each in multiple positions of two appendages.

The 224. multi-process exchange method as described in claim 223, wherein, the multiple position is built relative to the origin of coordinates It is vertical.

The 225. multi-process exchange method as described in claim 224, including：No matter the body overall positions and towards such as What, and in response to body action alternatively, using the absolute position and direction from by space, relative to the body The origin of coordinates is set up in the position that is selected in the group that fixed position and direction are constituted.

The 226. multi-process exchange method as described in claim 223, including the character in the multiple character is distributed into institute State each in the multiple directions of the first appendage.

The 227. multi-process exchange method as described in claim 221, wherein, the detection includes：Detect the outer of the body When the position pushed away intersects with Virtual Space, wherein, the Virtual Space, which is included in, is coupled to the aobvious of at least one processing equipment Show the space described in equipment.

The 228. multi-process exchange method as described in claim 227, including：When the position and the Virtual Space of the extrapolation During intersection, the virtual objects in the Virtual Space are controlled.

The 229. multi-process exchange method as described in claim 228, wherein, the control is included in the Virtual Space, In response to the position of the extrapolation, the position of the virtual objects in the Virtual Space is controlled.

The 230. multi-process exchange method as described in claim 228, wherein, the control includes：In response to the posture, control Make the situation of the virtual objects in the Virtual Space.

The 231. multi-process exchange method as described in claim 228, including：The control detection and the ratio of control, with life Into the uniformity between Virtual Space and physical space, wherein, the Virtual Space includes the space described over the display, its In, the physical space includes the space that the body is occupied.

The 232. multi-process exchange method as described in claim 231, including：In response at least one physical object in the thing The movement in space is managed to control at least one virtual objects in the Virtual Space.

The 233. multi-process exchange method as described in claim 201, including：It is described using the attitude signal control assembly Component is coupled at least one processing equipment.

The 234. multi-process exchange method as described in claim 233, wherein, control the component to include：By by the appearance State is mapped to three spatial objects simultaneously to control three spatial objects by six-freedom degree.

The 235. multi-process exchange method as described in claim 233, wherein, control the component to include：Pass through three translations The free degree and three rotary freedoms control three spatial objects.

The 236. multi-process exchange method as described in claim 235, wherein, three spatial object is being coupled at least one Show on the display device of processing equipment.

The 237. multi-process exchange method as described in claim 235, wherein, three spatial object is coupled to computer Remote system.

The 238. multi-process exchange method as described in claim 235, including：By the way that the posture is mapped into three space Multiple objects of object translate to control the motion of three spatial object.

The 239. multi-process exchange method as described in claim 238, wherein, it is described mapping include the posture with it is the multiple Direct mapping between object translation.

The 240. multi-process exchange method as described in claim 238, wherein, it is described mapping include the posture with it is the multiple Indirect mappers between object translation.

The 241. multi-process exchange method as described in claim 201, association is passed through using the data capsule and the multiple pond The operation of each in the multiple process is adjusted to form interactive application program from the multiple process.

The 242. multi-process exchange method as described in claim 201, including：It is described to coordinate using at least one in following The operation of multiple processes:The data capsule and the multiple pond.

The 243. multi-process exchange method as described in claim 201, including：Application program is divided into the set of process, its In, the multiple process includes the set of the process.

The 244. multi-process exchange method as described in claim 201, including by alternatively handling in the multiple pond extremely The data capsules of multiple retrievals in a few pond generates output process.

The 245. multi-process exchange method as described in claim 201, wherein, the multiple process includes multiple application programs Separable program performs context, wherein, each application program includes at least one process.

The 246. multi-process exchange method as described in claim 201, including：The multiple process is performed parallel.

The 247. multi-process exchange method as described in claim 201, including：The first set of parallel executive process, and it is suitable The second set of sequence executive process, wherein, the second of first set and the process of the multiple process including the process Set.

The 248. multi-process exchange method as described in claim 201, wherein, the representations of events process input.

The 249. multi-process exchange method as described in claim 201, wherein, the representations of events process output.

The 250. multi-process exchange method as described in claim 201, wherein, the event includes user interface event.

The 251. multi-process exchange method as described in claim 201, wherein, the event includes graphical event.

The 252. multi-process exchange method as described in claim 201, wherein, the representations of events process status.

The 253. multi-process exchange method as described in claim 252, wherein, the process status represents the interaction of the process Function, wherein, the interactive function of the process is as the content exposure of the data capsule in the multiple process.

The 254. multi-process exchange method as described in claim 253, including：By the content of the data capsule to define The application programming interface API of multiple processes is stated, rather than defines by funcall the API.

The 255. multi-process exchange method as described in claim 254, wherein, the content of the data capsule is and application program It is unrelated and can be by the multiple progress recognizing.

The 256. multi-process exchange method as described in claim 227, wherein, at least one described processing equipment includes multiple places Manage equipment.

The 257. multi-process exchange method as described in claim 256, wherein, at least one process in the multiple process Run under the first set of at least one processing equipment of the first set in the multiple processing equipment, in the multiple process At least one process at least one processing equipment of the second set in the multiple processing equipment second set under transport OK.

The 258. multi-process exchange method as described in claim 201, wherein, the multiple process includes the first process.

The 259. multi-process exchange method as described in claim 258, wherein, the translation is included the thing of first process Part is transformed to include at least one data of the status information for the first process event data and event for specifying the event Sequence.

The 260. multi-process exchange method as described in claim 259, wherein, the first process event data and status information It is the specific data of type with type corresponding with the application program of first process.

The 261. multi-process exchange method as described in claim 260, wherein, the translation includes：By the data capsule shape As including at least one described data sequence, the data capsule has including at least one data sequence and application The data structure of the unrelated expression of program.

The 262. multi-process exchange method as described in claim 258, wherein, the multiple process includes the second process.

The 263. multi-process exchange method as described in claim 262, wherein, the translation includes：By second process State change event be transformed to include to specify the event the second process event data and the event status information extremely A few data sequence.

The 264. multi-process exchange method as described in claim 263, wherein, the second process event data and status information It is the specific data of type with type corresponding with the application program of second process.

The 265. multi-process exchange method as described in claim 264, wherein, the translation includes：By the data capsule shape As including at least one described data sequence, the data capsule has including at least one data sequence and application The data structure of the unrelated expression of program.

The 266. multi-process exchange method as described in claim 262, wherein, the identification process is second process, institute Stating retrieval includes：The data capsule of retrieval from the multiple pond, and perform in the data capsule of the identification Second process of the processing of appearance.

The 267. multi-process exchange method as described in claim 266, wherein, the content of the data capsule of the identification is to represent The data of the first state of a process information.

The 268. multi-process exchange method as described in claim 267, wherein, the translation includes：By the data of the identification The content conversion of capsule is at least one new data sequence, and at least one described new data sequence represents first process Event and second process event at least one.

The 269. multi-process exchange method as described in claim 268, wherein, at least one described new data sequence includes institute State at least one state of a process information in the first process and second process and specify the event data of the event.

The 270. multi-process exchange method as described in claim 269, wherein, in first process and second process The event data and status information of at least one process be have with first process and second process extremely The specific data of type of the corresponding type of application program of a few process.

The 271. multi-process exchange method as described in claim 270, wherein, the translation includes：By the data capsule shape As including at least one described new data sequence, the data capsule, which has, includes at least one described new data sequence The expression unrelated with application program data structure.

The 272. multi-process exchange method as described in claim 271, wherein, at least one is new described in the multiple process use Data sequence.

The 273. multi-process exchange method as described in claim 266, wherein, the content of the data capsule to the identification Processing include render Drawing Object, wherein, the Drawing Object is rendered on the display of at least one processing equipment.

The 274. multi-process exchange method as described in claim 273, wherein, it is described to render including directly rendering, wherein described Multiple processes are drawn directly into the graph layer of at least one processing equipment, wherein, the multiple pond is used for for the wash with watercolours Contaminate the coordination between the multiple process.

The 275. multi-process exchange method as described in claim 273, wherein, it is described to render including following multiple processes：

The multiple pond will be sent to including the data capsule for rendering order；And

Render order from the retrieval of the multiple pond is described, explain described in render order, and render order in response to described and drive The graph layer of at least one processing equipment.

The 276. multi-process exchange method as described in claim 273, wherein, it is described to render including following multiple processes：

It is rendered into pixel buffer；

Original frame data is sent to the multiple pond, the original frame data as to being rendered described in the pixel buffer and Produce；And

Retrieve the original frame data from the multiple pond, and combine the original frame data, with described in driving at least Used in the graph layer of one processing equipment.

The 277. multi-process exchange method as described in claim 201, including：

Detect the event of the multiple process；

Generation includes specifying at least one data sequence of the status information of the event data of the event and the event, its In, the event data and status information are that the type with type corresponding with the application program of at least one processing equipment is special Fixed data；And

Data capsule shape, which is turned into, includes at least one described data sequence, the data capsule have include it is described at least one The data structure of the expression unrelated with application program of data sequence.

The 278. multi-process exchange method as described in claim 277, wherein, at least one described data sequence of generation includes：

Generation includes the first respective data set of the first respective event data；

Generation includes the second respective data set of the second respective status information；And

First data sequence is formed as to include the described first respective data set and the second respective data set.

The 279. multi-process exchange method as described in claim 278, wherein, generating the first respective data set includes：Will The first respective data set is formed as including the mark data of at least one processing equipment, and the mark data includes mark Know the data of at least one processing equipment.

The 280. multi-process exchange method as described in claim 278, wherein, at least one described data sequence of generation includes：

Generation includes the first respective data set of the first respective event data；

Generation includes the second respective data set of the second respective status information；And

Second data sequence is formed as to include the described first respective data set and the second respective data set.

The 281. multi-process exchange method as described in claim 280, wherein, generating the first respective data set includes：It is raw Into the first respective data set offset, wherein, the first respective data set offset points to described the of second data sequence One respective data set.

The 282. multi-process exchange method as described in claim 280, wherein, generating the second respective data set includes：It is raw Into the second respective data set offset, wherein, the second respective data set offset points to described the of second data sequence Two respective data sets.

The 283. multi-process exchange method as described in claim 278, wherein, the first respective data set is description chained list, The description chained list includes the description of data.

The 284. multi-process exchange method as described in claim 277, wherein, the event data is to represent typed data Labeling byte sequence.

The 285. multi-process exchange method as described in claim 284, wherein, the event data includes type header and type Specific data layout.

The 286. multi-process exchange method as described in claim 201, wherein, the status information is to represent typed data Labeling byte sequence.

The 287. multi-process exchange method as described in claim 286, wherein, the status information includes type header and type Specific data layout.

The 288. multi-process exchange method as described in claim 201, including：

Generate at least one skew；And

The data capsule shape, which is turned into, includes at least one described skew.

The 289. multi-process exchange method as described in claim 288, including：

First skew of the generation with the first variable-length；

Wherein, the event data of the first data sequence at least one data sequence is pointed in first skew.

The 290. multi-process exchange method as described in claim 288, including：

Second skew of the generation with the second variable-length；

Wherein, the status information of the first data sequence at least one data sequence is pointed in second skew.

The 291. multi-process exchange method as described in claim 288, including：

Pass through data capsule formation first code path using the first skew at least one described skew；

Pass through data capsule formation second code path using the second skew at least one described skew；

Wherein, the first code path and the second code path are different paths.

The 292. multi-process exchange method as described in claim 291, wherein, in first skew and the described second skew At least one includes metadata, and the metadata includes context certain metadata corresponding with the context of application program.

The 293. multi-process exchange method as described in claim 201, including：

Generation includes the head of the length of the data capsule；

The data capsule shape, which is turned into, includes the head.

The 294. multi-process exchange method as described in claim 201, including：The data capsule is sent to the multiple pond In pond.

The 295. multi-process exchange method as described in claim 294, including：

Detect the second event of at least one processing equipment；

Data capsule corresponding with the second event is searched in the multiple pond；

Recognize the correspondence between the data capsule and the second event；

The data capsule is extracted from the pond in response to the identification；And

In response to the content of the data capsule, at least one processing equipment execution is represented corresponding with the second event Processing operation, wherein, the application program of described at least one processing equipment and the first kind and the second application of Second Type Program correspondence.

The 296. multi-process exchange method as described in claim 294, wherein, multiple application programs are coupled in the multiple pond, The multiple pond includes multiple data capsules corresponding with the multiple application program, and the multiple pond is provided by the multiple Access of the application program to the multiple data capsule, wherein, at least two application programs in the multiple application program are Different application programs.

The 297. multi-process exchange method as described in claim 294, wherein, the multiple pond provides the shape of multiple data capsules State is cached.

The 298. multi-process exchange method as described in claim 294, wherein, the multiple pond provides the line of multiple data capsules Property sequence.

The 299. multi-process exchange method as described in claim 277, wherein, the data structure is non-typed.

The 300. multi-process exchange method as described in claim 201, wherein, the data structure of the data capsule provides described Event data and the status information are represented with platform-independent.

The 301. multi-process exchange method as described in claim 201, wherein, the data structure of the data capsule is provided to institute State being accessed with platform-independent for event data and the status information.

The 302. multi-process exchange method as described in claim 277, wherein, the transmission is included the data capsule from tool The first application program for having the first Application Type is sent at least one with least one the second Application Type Second application program, wherein, first Application Type is different from second Application Type, wherein, generate institute At least one data sequence is stated by first application program to perform, methods described includes：Kept during the transmission At least one data sequence of the data capsule is intact.

The 303. multi-process exchange method as described in claim 302, including make during the operation of second application program With at least one described data sequence.

The 304. multi-process exchange method as described in claim 201, including：The source that generation includes at least one processing equipment is set Standby event data and the first data set of mark data, the event data of the source device include specifying stepping on for the source device The data of the event of note, the mark data includes the data for identifying the source device.

The 305. multi-process exchange method as described in claim 304, including generation include the complete of the status information of the event Second data set of universal class, wherein, each in first data set and second data set is specific including type Data layout in typed data bundle.

The 306. multi-process exchange method as described in claim 305, wherein, the translation includes：By by data capsule shape As encapsulating first data set and second data set including first data set and second data set, its In, the data capsule has the data structure for the expression unrelated with application program for including at least one data sequence.

The 307. multi-process exchange method as described in claim 201, including：

Detect the event for the first processing equipment run under the application program of the first kind；

Generation includes the data sequence of the event data of first processing equipment, and the event data specifies the event and institute The status information of event is stated, wherein, the event data and status information are with type corresponding with the application program The specific data of type；

Data capsule shape, which is turned into, includes the data sequence, and the data capsule is with including the data sequence and application The data structure of the unrelated expression of program；

Detect the of the second processing equipment run under with the application program of at least one of at least one Second Type second Two events, wherein, the Second Type is different from the first kind, wherein, at least one processing equipment includes described first Processing equipment and the second processing equipment；

Recognize the correspondence between the data capsule and the second event；And

Operation is performed using the content of the data sequence of the data capsule, in response to the second event.

The 308. multi-process exchange method as described in claim 307, wherein, generating the data sequence includes：

Generation includes the first data set of the event data；

Generation includes the second data set of the status information；And

First data sequence is formed as to include first data set and second data set.

The 309. multi-process exchange method as described in claim 307, wherein, the event data is to represent typed data Labeling byte sequence.

The 310. multi-process exchange method as described in claim 309, wherein, the event data includes type header and type Specific data layout.

The 311. multi-process exchange method as described in claim 307, wherein, the status information is to represent typed data Labeling byte sequence.

The 312. multi-process exchange method as described in claim 311, wherein, the status information includes type header and type Specific data layout.

The 313. multi-process exchange method as described in claim 307, including：

Generate at least one skew；And

The data capsule shape, which is turned into, includes at least one described skew.

The 314. multi-process exchange method as described in claim 313, including：

First skew of the generation with the first variable-length, wherein, first skew is pointed at least one data sequence The event data of first data sequence；And

Second skew of the generation with the second variable-length, wherein, second skew is pointed at least one data sequence The status information of first data sequence.

The 315. multi-process exchange method as described in claim 313, including：

Pass through data capsule formation first code path using the first skew at least one described skew；

Pass through data capsule formation second code path using the second skew at least one described skew；

Wherein, the first code path and the second code path are different paths.

The 316. multi-process exchange method as described in claim 314, wherein, in first skew and the described second skew At least one includes metadata, and the metadata includes the specific first number of context corresponding with the context of the application program According to.

The 317. multi-process exchange method as described in claim 307, including the data capsule is sent to the multiple pond In pond.

The 318. multi-process exchange method as described in claim 317, including：

Data capsule corresponding with the second event is searched in the multiple pond；And

The data capsule is extracted in identification in response to the correspondence from the pond.

The 319. multi-process exchange method as described in claim 317, wherein, the multiple pond be coupled to the application program and At least one described second application program, the multiple pond include with the application program and it is described at least one second apply journey The corresponding multiple data capsules of sequence, the multiple pond, which is provided, passes through the application program and at least one described second application program Access to the multiple data capsule.

The 320. multi-process exchange method as described in claim 317, wherein, the multiple pond provides the shape of multiple data capsules State is cached.

The 321. multi-process exchange method as described in claim 317, wherein, the multiple pond provides the line of multiple data capsules Property sequence.

The 322. multi-process exchange method as described in claim 307, wherein, the data structure is non-typed.

The 323. multi-process exchange method as described in claim 307, wherein, the data structure of the data capsule provides described Event data and the status information are represented with platform-independent.

The 324. multi-process exchange method as described in claim 307, wherein, the data structure of the data capsule is provided to institute State being accessed with platform-independent for event data and the status information.

A kind of 325. multi-process interactive systems, including：

Detector, the attitude data for the posture that body is carried out is represented for receiving；And

Processor, is coupled to the detector, posture described in the processor from the attitude data automatic detection, the processing Device performs multiple processes, and the multiple process generation includes representing the event of the set of the event of the posture, the processor The event of each process in the multiple process is translated into data capsule, the data capsule is sent to by the processor Multiple ponds, wherein, the set running of the process in the multiple process is identification process, and the identification process is in the multiple pond Middle identification includes the data capsule of content corresponding with the posture, identification process retrieval from the multiple pond Data capsule, and by synthesize the identification data capsule content to form the attitude signal come from the identification Data capsule generates attitude signal, wherein, the attitude signal represents the posture,

Wherein, data capsule include initiating the event data of the state of a process information of the data capsule and event with application The unrelated expression of program.