JP7807092B2 - Electronic trading system and method based on point-to-point mesh architecture - Google Patents

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
This application is a continuation-in-part of U.S. Patent Application No. 16/988,510, filed August 7, 2020, and a continuation-in-part of U.S. Patent Application No. 16/988,491, filed August 7, 2020, the entire teachings of which are incorporated herein by reference.

Current financial trading systems allow traders to submit orders and receive confirmations, market data, and other information electronically over a communications network. The "electronic" marketplace enabled by what are sometimes called "electronic trading systems" has largely replaced the traditional pit-based "open outcry" trading system, in which traders or their representatives all physically reside in designated locations, i.e., trading pits, and trade with one another through verbal and visual/hand gesture communication, and has become the dominant means of trading most financial instruments.

When trading stocks and/or other financial instruments on an electronic trading system, the electronic trading system typically creates trades by matching trade orders, i.e., ask orders with corresponding bid orders. Depending on the conditions associated with the trade, the process of matching ask orders with bid orders can be complex.

One example of an electronic trading system is a computerized exchange that includes a central matching engine, typically residing in a central server, and multiple distributed servers or gateways. In such a computerized exchange, a typical process may be as follows: an order entry message, e.g., an ask order and/or a bid order, is sent from a client or participant device, e.g., a trader terminal, to the computerized exchange, which processes the order entry message. The order entry message is received at the central server via a gateway. Processing in the computerized exchange, e.g., at the central server, may include, among other things, performing order matching based on the received order entry message.

The order processing acknowledgement message generated by the central server is then typically transmitted back to the participant device via a gateway that forwards the transaction. The gateway may perform additional processing before transmitting the order processing acknowledgement message back to the participant device. The central server may also disseminate information about the order processing acknowledgement message, either in the same format as received or in another format, to one or more other gateways that perform processing of the order processing acknowledgement message to generate market data output via the market data stream. The market data output is typically forwarded to participant devices or other subscribers to the market data stream via various communication mechanisms and requires additional processing at the gateway.

According to an exemplary embodiment, an electronic trading system includes a gateway, a core compute node (interchangeably referred to herein as a core compute engine or a compute engine) configured to perform electronic trade matching functions, and a sequencer. The gateway and the core compute node are coupled via a first direct connection. The gateway and the sequencer are coupled via a second direct connection. The sequencer and the core compute node are coupled via a third direct connection. The first, second, and third direct connections have respective unshared bandwidths. The gateway is configured to send a message representing an electronic trade request with a limit price to buy or sell a financial instrument to the core compute node via the first direct connection. In response, the message is received by the core compute node. The gateway is further configured to send the message to the sequencer via the second direct connection, and the sequencer is configured to in turn send an order-marked message to the core compute node via the third direct connection. The sequencer is interposed between the gateway and the core compute node via the second and third direct connections. The order-marked messages sent by the sequencer are ordered versions of the messages sent by the gateway. In turn, the order-marked messages are received by a core compute node. The core compute node is configured to determine a relative ranking (i.e., order) of the order-marked messages among order-marked versions of other messages received by the core compute node in the electronic trading system. The core compute node is further configured to complete an electronic trade matching function for the electronic trade request responsive to the determined relative ranking, matching bids with counterparty bids for the financial instrument, and enabling electronic trading of the financial instrument.

The message and the sequence-marked message may contain the same user data. The user data is associated with the electronic transaction request.

The message may be a gateway message sent by the gateway in response to receiving an incoming message received by the gateway from a participant device. The sequencer is further configured to, in response, send an order-marked message to the gateway via the second direct connection. In response, the order-marked message is received by the gateway. The order-marked message is a first order-marked message. The core compute node is further configured to, in response to receiving the gateway message, send a core compute node message to the gateway via the first direct connection. The core compute node is further configured to send the core compute node message to the sequencer via the third direct connection, and the sequencer is further configured to, in response, send a second order-marked message to the gateway via the second direct connection. The second order-marked message is an ordered version of the core compute node message. The gateway is further configured to determine a relative ordering of the second order-marked message and the order-marked versions of other messages sent from the core compute node to the gateway. The gateway is further configured to send an outgoing message to the participant device. The outgoing message is transmitted according to the determined relative ranking. The sequencer is further configured to, in response, transmit the second order-marked message to the core compute node via the third direct connection.

The gateway may be a given gateway of the plurality of gateways. The core compute node may be a given core compute node of the plurality of core compute nodes. Each gateway of the plurality of gateways may be coupled to a respective core compute node of the plurality of core compute nodes via a respective first direct connection. The sequencer may be coupled to each gateway of the plurality of gateways via a respective second direct connection and to each core compute node of the plurality of core compute nodes via a respective third direct connection. The plurality of gateways, the plurality of core compute nodes, the sequencer, and their respective direct connections may constitute at least a portion of a point-to-point mesh system.

In a point-to-point mesh system, each gateway of the multiple gateways may be configured to transmit its respective compute node-destined message to all compute nodes and sequencers of the multiple core compute nodes. It should be understood that a message destined for a compute node may be interchangeably referred to herein as a "compute node-destined" message, and a message destined for a gateway may be interchangeably referred to herein as a "gateway-destined" message. Each core compute node of the multiple core compute nodes may be configured to transmit its respective gateway-destined message to all gateways and sequencers of the multiple gateways. The sequencers may be further configured to transmit their respective sequence-marked messages to the multiple gateways and multiple core compute nodes in response to receiving their respective compute node-destined messages or their respective gateway-destined messages.

The sequencer may be a given sequencer among a plurality of sequencers in the point-to-point mesh system. Each gateway of the plurality of gateways may be coupled to a respective sequencer of the plurality of sequencers via a respective second direct connection. Each core compute node of the plurality of core compute nodes may be coupled to a respective sequencer of the plurality of sequencers via a respective third direct connection. The given sequencer may be a currently active sequencer servicing the point-to-point mesh system. Each other sequencer of the plurality of sequencers may be a standby sequencer waiting to take over for the currently active sequencer. Each sequencer of the plurality of sequencers may be coupled to each other sequencer of the plurality of sequencers via a respective fourth direct connection. Each gateway of the plurality of gateways may be further configured to send messages destined for the respective compute node to a given sequencer among the plurality of sequencers. Each core compute node of the plurality of core compute nodes may be further configured to send messages destined for the respective gateway to a given sequencer among the plurality of sequencers. A given sequencer may be further configured to send an order-marked message to each other sequencer of the plurality of sequencers via each respective fourth direct connection to enable a standby sequencer to take over for the currently active sequencer if the currently active sequencer fails.

A message destined for each compute node sent by a given gateway may be the same as a message received by multiple core compute nodes. The multiple core compute nodes may be configured to generate response messages in response to receiving the same message. The response messages may be received at a given gateway from among the multiple core compute nodes. The given gateway may be further configured to take action based on a given response message from among the response messages generated in response to receiving the same message. The given response message may arrive at the given gateway first relative to other response messages generated in response to receiving the same message. The gateway may be further configured to ignore other response messages that arrive after the given response message.

Multiple compute node-destined messages representing the same message may be received from multiple gateways at a given compute node. The given compute node may be further configured to take action based on a message destined for the given compute node among the multiple compute node-destined messages. A message destined for the given compute node may arrive at the given compute node first relative to messages destined for other compute nodes among the multiple compute node messages representing the same message. A given core compute node may be further configured to ignore messages destined for other compute nodes that arrive after the message destined for the given compute node.

The electronic trading system may further include an order book accessible by the core compute node. The core compute node may be further configured to match trading orders for financial instruments using an electronic trade matching function. The core compute node may be further configured to maintain open positions for financial instruments in the order book. A price discrepancy for a financial instrument may result from executing the electronic trade matching function. The open positions may include price discrepancies for financial instruments. It should be understood that open positions can convey more information than quantity. For example, positions may be bullish and bearish (sides). According to an example embodiment, the open positions convey both price and quantity.

The electronic trading system may further include a clock. The gateway, core compute nodes, and sequencer may be synchronized based on the clock.

The gateway may be further configured to serve at least one participant device and to send messages to the sequencer and the core compute node in response to receiving an incoming message at the gateway. The incoming message may be dispatched by at least one participant device. The sequencer may be further configured to generate an order-marked message by marking the message with a unique order identifier or by creating a representation of the received message, marking the representation with the unique order identifier, and transmitting the marked representation. The marked representation may be an order-marked message.

The electronic trading system may further include at least one redundant direct connection for each of the first direct connection, the second direct connection, and the third direct connection, or some set thereof.

The gateway may be a given gateway among a plurality of gateways communicatively coupled to each other via a shared gateway network. The core compute node may be a given core compute node among a plurality of core compute nodes communicatively coupled to each other via a shared core compute node network. The sequencer may be a given sequencer among a plurality of sequencers communicatively coupled to each other via a shared sequencer network or via respective fourth direct connections.

The electronic trading system may further include a system status log. A given sequencer may be configured to transmit the system status log to at least one other sequencer in the plurality of sequencers via a shared sequencer network, or to store the system status log in a data store. The data store may be accessible to the plurality of sequencers via the shared sequencer network.

The electronic trading system may be an active electronic trading system, and at least one sequencer of the plurality of sequencers may be communicatively coupled to a disaster recovery site. The disaster recovery site may include a standby electronic trading system. The standby electronic trading system may be a replica of the active electronic trading system and may be configured to allow electronic trading to continue if the active electronic trading system fails.

The gateway, core compute node, sequencer, and first, second, and third direct connections may constitute a first point-to-point mesh system. The electronic trading system may be a first electronic trading system communicatively coupled to a proxy node. The proxy node may be further communicatively coupled to at least one participant device and a second electronic trading system. The second electronic trading system may include a second point-to-point mesh system. The proxy node may be configured to send messages to the first and second electronic trading systems in response to receiving an incoming message from at least one participant device. The first and second electronic trading systems may be configured to generate respective responses to the message sent by the proxy node. The proxy node may further be configured to send a response to at least one participant device in response to receiving a first-arriving response among the respective responses generated and received from the first or second electronic trading system.

According to another exemplary embodiment, a method for executing electronic trading comprises transmitting a message representing an electronic trade request with a limit price to buy or sell a financial instrument from a gateway to a core compute node via a first direct connection. The method further comprises, in response, receiving the message at the core compute node in an electronic trading system for executing an electronic trading function. The method further comprises transmitting the message from the gateway to a sequencer in the electronic trading system via a second direct connection, and in response, transmitting an order-marked message from the sequencer to the core compute node via a third direct connection. The first, second, and third direct connections have respective unshared bandwidths. The sequencer is inserted between the gateway and the core compute node via the second and third direct connections. The order-marked message transmitted by the sequencer is an ordered version of the message transmitted from the gateway via the second direct connection. The method further comprises, in response, receiving the order-marked message at the core compute node. The method further includes receiving, at the core compute node, other messages from the gateway and receiving, at the core compute node, order-marked versions of the other messages from the sequencer. The method further includes determining, at the core compute node, a relative ranking of the order-marked message among order-marked versions of other messages received by the core compute node in the electronic trading system. The method further includes completing, at the core compute node, an electronic trade matching function for the electronic trade request in response to the determined relative ranking, where the completing matches a limit price with a counterparty limit price for the financial instrument to enable electronic trading of the financial instrument.

Alternative method embodiments are similar to those described above in connection with the exemplary electronic trading system embodiments.

According to another exemplary embodiment, a sequencer of an electronic trading system comprises a non-transitory computer-readable medium encoding a set of instructions that, when loaded into and executed by the sequencer, causes the sequencer to communicate directly with a gateway via a first direct connection in a point-to-point mesh system of the electronic trading system. The set of instructions further causes the sequencer to communicate directly with a core compute node via a second direct connection in the point-to-point mesh system of the electronic trading system. The sequencer is interposed between the gateway and the core compute node via the first and second direct connections, the first and second direct connections having respective unshared bandwidths. The set of instructions further causes the sequencer to generate an order-marked message by marking a message or a representation thereof with a unique order identifier, the message being received by the sequencer via the first or second direct connection, respectively. The set of instructions further causes the sequencer to transmit the order-marked message to the gateway and the core compute node.

According to another exemplary embodiment, an electronic trading system includes a gateway coupled to a core compute node via an activation link, a ranking path, and a sequencer electronically disposed within the ranking path. The gateway is configured to send messages to the core compute node via the activation link and the ranking path. The core compute node is configured to receive messages and order-marked versions of the messages from the gateway and the sequencer, respectively. The order-marked versions include an order identifier indicating a deterministic position of the message among multiple messages communicated via the activation link and received by the sequencer via the ranking path. The messages and order-marked versions include common metadata.

The core compute node may be configured to (i) initiate matching function activities for an electronic transaction in response to receiving a message via the activation link, and (ii) use the order identifier to prioritize completion of matching function activities toward servicing the electronic transaction in response to receiving an order-marked version via the ranking path.

The message and the order-marked version may include common metadata. The core compute node may be further configured, upon receiving the order-marked version via the ordering path, to correlate the message to the order-marked version based on the common metadata.

According to yet another example embodiment, an electronic trading system may include a gateway, a sequencer, and a core compute node arranged in a point-to-point mesh topology. The core compute node may be configured to perform a matching function toward servicing trade requests received from participant devices and introduced into the point-to-point mesh topology via the gateway. The point-to-point mesh topology may include a first direct connection, a second direct connection, and a third direct connection. The sequencer may be configured to (i) determine a deterministic ranking (i.e., order) for messages communicated between the gateway and the core compute node via the first direct connection and received by the sequencer from the gateway or the core compute node via the second or third direct connection, respectively. The sequencer may be further configured to (ii) communicate the position of the message within the deterministic ranking by sending order-marked versions of the message representing the trade request or a response thereto to the gateway and the core compute node via the second and third direct connections, respectively.

It should be understood that the exemplary embodiments disclosed herein may be implemented in the form of a method, apparatus, system, or computer-readable medium having program code embodied therein.

The above will become apparent from the following more detailed description of exemplary embodiments, as shown in the accompanying drawings, in which like numerals refer to like parts throughout the various views. The drawings are not necessarily to scale, emphasis instead being placed on the embodiments illustrated.

FIG. 1A is a block diagram of an example embodiment of a market for trading financial instruments. FIG. 1B-1 is a block diagram of an example embodiment of an electronic trading system. FIG. 1B-2 is a block diagram of an example embodiment of another electronic trading system. FIG. 1C is a block diagram of an example embodiment of a point-to-point mesh system. FIG. 1D is a block diagram of another example embodiment of an electronic trading system. FIG. 1E is a table of an example embodiment of fields of a message format for a transaction message. FIG. 1F is a flow diagram illustrating an example embodiment of the operation of an electronic trading system. FIG. 1G is a flow diagram illustrating another example embodiment of the operation of an electronic trading system. FIG. 2 is a block diagram of an example embodiment of a mesh node in a point-to-point mesh architecture of an electronic trading system. FIG. 3 is a block diagram of another example embodiment of a point-to-point mesh system. FIG. 4 is a block diagram of another example embodiment of a point-to-point mesh system. FIG. 5 is a flow diagram of an example embodiment of a method for conducting an electronic transaction. FIG. 6 is a block diagram of an exemplary embodiment of a sequencer for an electronic trading system. FIG. 7 is a block diagram of another example embodiment of an electronic trading system.

An example embodiment is described below.

It should be understood that the dedicated/direct connections disclosed herein are point-to-point connections that do not pass through a shared network switch.

While current electronic trading systems attempt to provide performance advantages, many suffer from performance degradation due to the increasing transaction volume resulting from an increasing number of market participants. Implementations by some market participants are often based on high-frequency trading methods in which high-speed computers automatically monitor the market and react to market events, usually in an overwhelming manner. Additionally, there is a continuing demand for ever-decreasing processing latency and response times, resulting in a need for further capacity and performance improvements to maintain the performance experienced by each market participant and avoid deleterious consequences such as capacity starvation and unfair access.

Increasing the speed at which market participants can assess and respond to changes in market data, such as responding to market events, increases the rate at which transactions are received by electronic trading systems, shortens the time between receipts, and necessitates the need for more sophisticated discrimination to determine the order (i.e., sequence) in which those transactions are received. This, for example, is the basis for the deterministic operation of electronic trading systems, such as for trade order allocation. Furthermore, adding channels of communication to electronic trading systems allows more transactions to be submitted to electronic trading systems via multiple parallel paths, increasing capacity and opportunity with increasing bandwidth for each channel.

It is therefore useful for electronic trading systems to identify incoming transactions received within a short time frame. It is even more useful for such systems to reconcile transactions that were received simultaneously or so close in time that they are considered to have been received simultaneously. In addition to increased capacity and lower latency, the global nature of business further drives the need for fault tolerance, which increases the availability and reliability of electronic trading systems.

A business transaction may be defined as one or more actions or operations undertaken in accordance with one or more relevant business rules (including industry, legal, or regulatory requirements or practices) to achieve a business or commercial objective, which may include compliance with industry, regulatory, or legal requirements. A business transaction may be implemented by one or more computer processes and/or database operations/program actions, which may themselves be referred to as transactions. Business transactions, as defined by the relevant business rules, may be characterized as deterministic in that they may be characterized by interdependencies or relationships that affect their outcomes, such as dependencies on the order (i.e., sequence) in which they are processed, such as temporal ordering, and/or dependencies on real-time processing, as defined by the business rules, to enable the business/commercial objective and/or meet participant expectations. This is referred to herein as "transaction determinism." Generally, a set of deterministic transactions will produce a particular result when executed in one order (i.e., sequence), and will produce a different result when executed in a different order (i.e., sequence). In some applications, deterministic processing may be preferred/prioritized over real-time processing.

It is useful for high-performance electronic trading systems to ensure transaction determinism under increasing loads while providing improved trading opportunities, fault tolerance, low-latency processing, high capacity (e.g., processing many messages per second), risk mitigation and market protection with minimal impact, and fair access to information and opportunities.

The exemplary embodiments disclosed herein relate to a high-speed electronic trading system that provides a market in which orders to buy and sell financial instruments, such as stocks, bonds, commodities, futures, and options, are traded between market participants, such as traders and brokers. The exemplary embodiments of the electronic trading system disclosed herein exhibit low latency, fairness, fault tolerance, and other features more fully described below.

FIG. 1A is a block diagram of an example embodiment of a market 90, including an example embodiment of an electronic trading system 100 used to trade financial instruments (not shown). The electronic trading system 100 is primarily responsible for "matching" trading orders with one another and employs a point-to-point mesh system 102 to do so. In one example, a limit order to "buy" a financial instrument is matched by the matching engine of the point-to-point mesh system 102 to a corresponding counter limit order to "sell" the financial instrument. The matched limit order and counter limit order must at least partially satisfy the desired price, with any remaining unfilled quantity being passed on to other appropriate counterorders. The matched trading orders are then mated and the trade is executed.

Unfilled or partially filled orders are maintained in a data structure called an "order book" (not shown). Pending information about unmatched trade orders is available to the matching engine to fill subsequent trade orders. An order book is typically maintained for each financial instrument and generally defines or represents the state of the market 90 for that particular instrument, i.e., for that particular financial instrument. The order book may include, for example, recent prices and quantities at which market participants expressed their willingness to buy or sell.

The results of the matching may be made visible to market participants via a streaming data service (not shown) called a market data feed (not shown). A market data feed typically includes individual messages conveying pricing for each financial instrument traded as well as related information such as volume and other statistics.

In market 90, market participants include two traders, namely, first trader 104a and second trader 104b. It should be understood that market 90 is not limited to market participants who are traders, and market 90 is not limited to two traders. In market 90, market participants such as first trader 104a and second trader 104b may submit trade orders and receive confirmations, market data, and other information electronically via a communications network (not shown).

In the exemplary embodiment of FIG. 1A, a first trader 104a submits a first trade order (not shown) to buy a financial instrument (not shown) via a first incoming message 3a transmitted to the electronic trading system 100 via a first participant device 130a. The electronic trading system 100 employs a point-to-point mesh system 102 that matches the first trade order against a second trade order (not shown) to sell the financial instrument. A second trade order to sell the financial instrument is submitted by a second trader 104b via a second incoming message 3b transmitted to the electronic trading system 100 via a second participant device 130b.

In an exemplary embodiment, the electronic trading system 100 sends a first outgoing message 5a and a second outgoing message 5b to the first participant device 130a and the second participant device 130b, respectively, to notify the first trader 104a and the second trader 104b of the successful execution of their respective trade orders. The point-to-point mesh system 102 enables the electronic trading system 100 to execute high-speed, deterministic electronic trades of financial instruments. An exemplary embodiment of the point-to-point mesh system 102 is disclosed below with reference to Figures 1B-1 and 1B-2.

FIG. 1B-1 is a block diagram of an exemplary embodiment of the electronic trading system 100 of FIG. 1A disclosed above. In a particular embodiment, the electronic trading system 100 includes a gateway 120-1 coupled to a core compute node 140-1 via an activation link 180-1-1 and a ranking (i.e., ordering) path 117. The electronic trading system 100 further includes a sequencer 150-1 electronically located within the ranking path 117. The gateway 120-1 is configured to send a message (not shown) to the core compute node 140-1 via the activation link 180-1-1 and the ranking path 117. The message may be a message 106 disclosed below with respect to FIG. 1B-2. The core compute node 140-1 is configured to receive the message and an order-marked version of the message (not shown) from the gateway 120-1 and the sequencer 150-1, respectively. The order-marked version of the message may be the order-marked message 106′ of FIG. 1B-2, as further disclosed below. The order-marked version (i.e., the order-marked message 106′) includes an order identifier (ID), such as may be included in the order ID field 110-14 of the order-marked message 106′, as further disclosed below with respect to FIG. 1E for a non-limiting example. The order ID indicates a deterministic position of the order-marked version of the message among order-marked versions of other messages communicated over activation link 180-1-1 and received by sequencer 150-1 via ordering path 117. The messages for which the order ID indicates a deterministic position also include order-marked versions of other messages received by core compute node 140-1 via ordering path 117. The message (e.g., an unordered message or a non-order-marked message) and the order-marked version include common metadata (not shown). As further disclosed below, the order ID of the message is identified by correlating the message to its order-marked version via the common metadata. The sequence ID further indicates the definitive position of the message among all messages communicated throughout the electronic trading system 100 that have passed through, and are consequently sequence-marked by, sequencer 150-1.

It should be understood that while elements of electronic trading system 100 timestamp messages communicated therein, the sequence ID determined by sequencer 150-1 determines the position (rank/priority) of a message communicated in electronic trading system 100. It is possible for multiple systems to timestamp a message with the same timestamp, and as a result, the rank/priority for that message may have to be determined at the recipient's end. This does not occur in electronic trading system 100, as sequencer 150-1 can be the sole entity that determines the rank/priority of messages communicated throughout electronic trading system 100.

Core compute node 140-1 may be configured to (i) initiate matching function activities for an electronic transaction in response to receiving a message via activation link 180-1-1, and (ii) use an order identifier to prioritize completion of matching function activities toward servicing the electronic transaction in response to receiving an order-marked version via ranking path 117.

The message and the order-marked version may include common metadata. Core compute node 140-1 may be further configured, upon receiving the order-marked version via ordering path 117, to correlate the message with the order-marked version based on the common metadata.

In the exemplary embodiment of FIG. 1B-1, the message is sent in the activation link forward direction, i.e., at act-link-fwd-dir 113a, via activation link 180-1-1 to core compute node 140-1, and in the ordering path forward direction, i.e., at order-path-fwd-dir 115a, via ordering path 117 to core compute node 140-1. Following completion of the matching function activity, core compute node 140-1 may send a response (not shown) in the activation link reverse direction (i.e., act-link-rev-dir 113b) and ordering path reverse direction (i.e., order-path-rev-dir 115b) to gateway 120-1 via activation link 180-1-1 and ordering path 117. The response sent to gateway 120-1 via activation link 180-1-1 and ordering path 117 may be response 107, which is further disclosed below with respect to FIG. 1B-2.

Activation link 180-1-1 may be a single direct connection, while ordering path 117 may include multiple direct connections. For example, activation link 180-1-1, also referred to as first direct connection 180-1-1, may be a single direct connection, while ordering path 117 may include second direct connection 180-gw1-s1 and third direct connection 180-c1-s1, as further disclosed below with respect to FIG. 1B-2.

Gateway 120-1, sequencer 150-1, and core compute node 140-1 are arranged in a point-to-point mesh topology. Core compute node 140-1 may be configured to perform a matching function (i.e., an electronic trade matching function) toward servicing trade requests received from participant devices and introduced into the point-to-point mesh topology via gateway 120-1, as further disclosed below with respect to FIG. 1B-2. The point-to-point mesh topology includes a first direct connection, a second direct connection, and a third direct connection, as disclosed above and further disclosed below with respect to FIG. 1B-2. Sequencer 150-1 may be configured to (i) determine a deterministic ranking (i.e., order) for messages communicated between gateway 120-1 and core compute node 140-1 via the first direct connection and received by sequencer 150-1 from gateway 120-1 or core compute node 140-1 via the second or third direct connection, respectively. Sequencer 150-1 may further be configured to (ii) communicate the position of the message within the deterministic order by sending an order-marked version of the message to gateway 120-1 and core compute node 140-1 via the second and third direct connections, respectively. The message may represent a transaction request or a response thereto, as disclosed below with respect to FIG. 1B-2.

FIG. 1B-2 is another block diagram of an exemplary embodiment of the electronic trading system 100 of FIG. 1A disclosed above. The electronic trading system 100 includes a gateway 120-1, a core compute node 140-1 configured to perform electronic trade matching functions, and a sequencer 150-1. The gateway 120-1 and the core compute node 140-1 are coupled via a first direct connection 180-1-1. The gateway 120-1 and the sequencer 150-1 are coupled via a second direct connection 180-gw1-s1. The sequencer 150-1 and the core compute node 140-1 are coupled via a third direct connection 180-c1-s1. The first direct connection 180-1-1, the second direct connection 180-gw1-s1, and the third direct connection 180-c1-s1 each have their own unshared bandwidth.

In some figures of the disclosure, arrows are included on direct connections (i.e., direct links), such as the arrows on the first direct connection 180-1-1, second direct connection 180-gw1-s1, and third direct connection 180-c1-s1 shown in FIG. 1B-2. The arrows on such direct connections/links indicate that data may flow bidirectionally through the direct connections/links. While other figures of the disclosure, such as FIG. 1B-1 disclosed above and FIG. 1D further disclosed below, may not have the above arrows applied to direct connections/links, it should be understood that data may flow bidirectionally along such connections/links, and that such connections/links may be a single communication link or two parallel links, each of which is unidirectional and data flows in opposite directions through the two links.

Continuing with reference to FIG. 1B-2, gateway 120-1 is configured to transmit message 106 (i.e., B) representing an electronic transaction request with a limit price to buy or sell a financial instrument to core compute node 140-1 via first direct connection 180-1-1, whereby message 106 is received by core compute node 140-1 in return. Gateway 120-1 is further configured to transmit message 106 (i.e., B) to sequencer 150-1 via second direct connection 180-gw1-s1, whereby sequencer 150-1 is configured to transmit sequence-marked message 106' (i.e., C) to core compute node 140-1 via third direct connection 180-c1-s1, as further disclosed below with respect to FIG. 1D. Sequencer 150-1 is interposed between gateway 120-1 and core compute node 140 via the second and third direct connections. The order-marked message 106' sent by sequencer 150-1 is an ordered version of the message sent by gateway 120-1. In turn, the order-marked message is received by core compute node 140-1. Core compute node 140-1 is configured to determine a relative ranking of the order-marked message 106' among order-marked versions of other messages (not shown) received by core compute node 140-1 in electronic trading system 100. Core compute node 140-1 is further configured to complete an electronic trade matching function for the electronic trade request in response to the determined relative ranking, thereby enabling electronic trading of the financial instrument by matching the limit price with a counterparty limit price for the financial instrument.

It should be understood that the electronic trade matching function may include more than matching trade orders per se. For example, the electronic trade matching function may include sending an acknowledgement message, as further disclosed below with respect to FIG. 1D. Core compute node 140-1 may also perform a portion of the electronic trade matching function prior to receiving an order-marked message 106' that allows core compute node 140-1 to facilitate it. For example, following receipt of message 106 (e.g., an unordered message), core compute node 140-1 may initiate the electronic trade matching function by loading data related to the stock symbol identified in message 106 into high-speed memory for later access, such as, for non-limiting example, the stock symbol in stock symbol field 110-2, further disclosed below with respect to FIG. 1E. The stock symbol may represent a traded security, such as a stock symbol or stock ticker. However, it should be understood that core compute node 140-1 may initiate (trigger) the electronic trade matching function by performing any type of activity related to the content of message 106.

Sequencer 150-1 may be further configured to generate sequence-marked message 106' by marking message 106 or a representation thereof with a unique sequence identifier (not shown). The unique sequence identifier may have a value corresponding to the arrival time of message 106 at sequencer 150-1 and may indicate the relative sequence position of message 106 among multiple messages (not shown) received at sequencer 150-1.

While core compute node 140-1 may initiate electronic trading functions and thereby begin processing unordered message 106 in response to receiving unordered message 106, as described above, core compute node 140-1 may not complete processing of message 106 and/or may not commit to the results of processing message 106 until core compute node 140-1 receives order-marked message 106'. Without a deterministic ordering for processing messages, such as specified via an order identifier in order-marked message 106', for example, processing of messages by compute node 140-1 may be unpredictable. As a non-limiting example of a possible unpredictable outcome, there may be multiple outstanding unordered messages, each representing a potential match for a counterparty in an exchange of securities. A deterministic manner of arbitrating among multiple potential matches is useful, since likely only a subset of potential matches will be executable for a given trade order of the counterparty.

In some embodiments, after receiving both the unordered message 106 and the order-marked message 106', compute node 140-1 may correlate the unordered message 106 with the order-marked message 106' via identifying information in both versions of the message, as described below in connection with FIG. 1E. Once compute node 140-1 receives the order-marked message 106', compute node 140-1 may then determine the appropriate order in which message 106/106' should be processed relative to other messages throughout electronic trading system 100 and may complete processing of message 106/106', including sending appropriate response messages and possibly referencing the order identifier assigned by the sequencer to order-marked message 106'. Returning to the non-limiting example of multiple messages representing potential matches to counterparties in an exchange of securities, when order-marked versions of the messages representing potential matches are received by compute node 140-1, compute node 140-1 may determine the exact order in which the potential matches occur to complete the electronic trade matching function.

According to an example embodiment, in addition to transmitting the order-marked message 106' to compute node 140-1 via the third direct connection 180-c1-s1, sequencer 150-1 may also transmit the order-marked message 106' (i.e., C) to gateway 120-1 via the second direct connection 180-gw1-s1. Providing the order-marked message 106' (i.e., C) to the sender of message 106 allows the sender, i.e., gateway 120-1, to correlate the sequence number assigned to the message, i.e., message 106, with other identifying information in the message (as described below in connection with FIG. 1E). This allows the sender to easily address subsequent messages that reference that sequence number, as further disclosed below with respect to FIG. 1D.

Gateway 120-1, core compute node 140-1, sequencer 150-1, first direct connection 180-1-1, second direct connection 180-gw1-s1, and third direct connection 180-c1-s1 constitute a point-to-point mesh system 102. According to an example embodiment, in point-to-point mesh system 102, first direct connection 180-1-1, second direct connection 180-gw1-s1, and third direct connection 180-c1-s1, or some combination thereof, may be protected by at least one respective redundant direct connection (not shown). If this direct connection fails, the respective redundant direct connection may be employed instead. In this manner, electronic trading system 100 may further include at least one respective redundant direct connection for each of the first, second, and third direct connections, or some combination thereof.

According to an example embodiment, the electronic trading system 100 may further include a clock, and the gateway 120-1, core compute node 140-1, and sequencer 150-1 may be synchronized based on a clock such as clock 195 in FIG. 1D, as further disclosed below.

Message 106 may be referred to as a "gateway" message because it arrives from a gateway, i.e., gateway 120-1. Message 106 may also be referred to as a "compute node-destined" message because, in the illustrated embodiment, it is destined for a compute node, i.e., core compute node 140-1. Message 106 is a gateway message sent by gateway 120-1 in response to receipt of incoming message 103 (i.e., A) received by gateway 120-1 from a participant device (not shown). Order-marked message 106' may be the first order-marked message. Core compute node 140-1 may be further configured, in response to receiving message 106, i.e., gateway message, to send a core compute node message, i.e., response 107 (i.e., D), to gateway 120-1 via first direct connection 180-1-1 and to send the core compute node message (i.e., response 107) to sequencer 150-1 via third direct connection 180-c1-s1. Sequencer 150-1 is further configured, in response, to send a second order-marked message, i.e., order-marked response 107' (i.e., E), to gateway 120-1 via second direct connection 180-gw1-s1. The second order-marked message is an ordered version of the core compute node message. Gateway 120-1 may be further configured to determine a relative ranking of the second order-marked message (i.e., order-marked response 107') and order-marked versions of other messages sent from core compute node 140-1 to gateway 120-1. Gateway 120-1 may be further configured to send outgoing message 105 (i.e., F) to participant devices according to the determined relative ranking. It should be understood that message 106 and response 107 relate to trading activity.

Sequencer 150-1 may further transmit a second order-marked message, i.e., order-marked response 107' (i.e., E), to core compute node 140-1 via third direct connection 180-c1-s1. By providing order-marked response 107' (i.e., E) to the sender of response 107, the sender, i.e., core compute node 140-1, can correlate the sequence number assigned to the message, i.e., response 107, with other identifying information in the message (as described below in connection with FIG. 1E). This allows the sender to easily address subsequent messages that reference that sequence number, as further disclosed below with respect to FIG. 1D.

Similar to the above with respect to messages 106 and 106' received by compute node 140-1, in response to receiving the unordered response message 107, gateway 120-1 may activate processing of the response message even before gateway 120-1 receives the order-marked response 107'. By way of non-limiting example, activating (starting) processing may include updating the state of gateway 120-1's open trade order database and/or storing the outgoing message 105 as available for transmission to a participant device. However, in some embodiments, gateway 120-1 may not complete processing of the order-marked response message 107 (which processing may include transmitting the outgoing message 105 to a participant device) until gateway 120-1 receives an order-marked response message 107' that includes an order identifier specifying the definitive position of the response message 107 in a sequence of messages, including other messages, in electronic trading system 100. In some embodiments, after receiving both the unordered message 107 and the order-marked reply message 107', the gateway 120-1 may correlate the unordered reply message 107 with the order-marked reply message 107' via identifying information in both versions of the messages, as described below in connection with FIG. 1E. This allows the definitive location of the reply message 107/107' to be determined upon receipt of the order-marked reply message 107'. In some embodiments, processing of the reply message may then be completed, which may include committing the outgoing message 105 to be sent to the participant device.

Electronic trading system 100 may further include an order book (not shown) accessible by core compute node 140-1. Core compute node 140-1 may be further configured to match trade orders for financial instruments (not shown) based on an executed electronic trade matching function. For example, core compute node 140-1 may be further configured to match trade orders for financial instruments using the electronic trade matching function. Core compute node 140-1 may be further configured to maintain a balance (not shown) of the financial instruments in the order book. A price discrepancy for the financial instruments may result from executing the electronic trade matching function. The balance may include a price discrepancy for the financial instruments. It should be understood that the balance can convey more information than quantity. For example, the balance may be bullish and bearish (side). According to an example embodiment, the balance conveys both price and quantity.

Gateway 120-1 may further be configured to serve at least one participant device (not shown) and to transmit message 106 to sequencer 150-1 and core compute node 140-1 in response to receiving an incoming message 103 at gateway 120-1. The incoming message 103 is dispatched by at least one participant device. Sequencer 150-1 may further be configured to generate an order-marked message by marking the message with a unique order identifier or by creating a representation of the received message, marking the representation with the unique order identifier, and transmitting the marked representation. The marked representation may be an order-marked message.

According to an example embodiment, gateway 120-1 may be a given gateway among a plurality of gateways, such as the plurality of gateways of Figures 1C, 1D, 3, and 4, further disclosed below. Also, core compute node 140-1 may be a given core compute node among a plurality of core compute nodes, such as the plurality of core compute nodes of Figures 1C, 1D, 3, and 4, further disclosed below.

1C is a block diagram of an example embodiment of a point-to-point mesh system 122. The point-to-point mesh system 122 includes a plurality of gateways 120, a plurality of core compute nodes 140, and a sequencer 150-1. Each gateway 120-1, 120-2, ..., 120-g of the plurality of gateways 120 is coupled to a respective core compute node 140-1, 140-2, ..., 140-c of the plurality of core compute nodes 140 via a respective first direct connection, i.e., first direct connection 180-a. The sequencer 150-1 is coupled to each gateway of the plurality of gateways via a respective second direct connection, i.e., second direct connection 180-b, and is coupled to each core compute node of the plurality of core compute nodes via a respective third direct connection, i.e., third direct connection 180-c. The multiple gateways, the multiple core compute nodes, the sequencer 150-1, and their respective direct connections form at least a portion of the point-to-point mesh system 122.

In the point-to-point mesh system 122, each gateway of the plurality of gateways 120 is configured to transmit its respective compute node-destined message to all of the plurality of core compute nodes 140 and to the sequencer 150-1. In the point-to-point mesh system 122, each core compute node of the plurality of core compute nodes 140 is configured to transmit its respective gateway-destined message to all of the plurality of gateways 120 and to the sequencer 150-1. In the point-to-point mesh system 122, the sequencer 150-1 is further configured to transmit its respective order-marked message to the plurality of gateways 120 and the plurality of core compute nodes 140 in response to receiving the respective compute node-destined message or the respective gateway-destined message.

Each compute node-destined message sent by a given gateway among the plurality of gateways 120 is the same message received by the plurality of core compute nodes 140. At least two compute nodes 140 among the plurality of core compute nodes 140 may be configured to generate a response message in response to receiving the same message. The response message may be received at a given core compute node among the plurality of core compute nodes. The given gateway may be further configured to take action based on a given response message among the response messages generated in response to receiving the same message. The given response message may arrive at the given gateway first relative to other response messages generated in response to receiving the same message. The given gateway may be further configured to ignore other response messages that arrive after the given response message. Such response messages may be referred to as functionally equivalent messages. Functionally equivalent messages refer to the same message that causes a recipient of the functionally equivalent message to perform the same function (e.g., one or more activities). Functionally equivalent messages represent the same message and may indeed be identical in some embodiments, although in other embodiments, functionally equivalent messages may not necessarily be identical, possibly including different metadata, such as, by way of non-limiting example, different timestamps, different source identifiers, etc. For clarity, the term "functionally equivalent message" does not refer to a non-order-marked message that is related to its order-marked message counterpart. Instead, although there may be multiple order-marked messages that represent the same order-marked message in some embodiments, functionally equivalent messages generally refer to multiple non-order-marked messages that represent the same non-order-marked message, and thus we may refer to functionally equivalent non-order-marked messages or functionally equivalent order-marked messages. The following description of FIG. 1E describes at least one example of how a recipient node may determine that two or more such response messages are functionally equivalent and represent the same message.

Multiple functionally equivalent messages (not shown) may be received at a given gateway from multiple core compute nodes 140, multiple sequencers 150, or a combination thereof. A given gateway may be further configured to take action based on a given functionally equivalent message from the multiple functionally equivalent messages, where the given functionally equivalent message is the first to arrive at the given gateway. A given gateway may be further configured to ignore other functionally equivalent messages from the multiple functionally equivalent messages that arrive after the given functionally equivalent message. Such messages may be understood to be "functionally" equivalent because, for the same message received at multiple core compute nodes, each of the multiple core compute nodes independently generates a response message, so that each respective response achieves the same functional result, even if it is not exactly identical. For example, such response messages may have at least a different originating core identifier included therein to uniquely identify the particular core compute node sending the response.

According to an example embodiment, the at least two "functionally equivalent messages" may arrive at a given core compute node, such as core compute node 140-1, from a gateway/sequencer. The given core compute node may be configured to process only the first of such functionally equivalent messages to arrive. In this manner, multiple functionally equivalent messages may be received at a given compute node from multiple gateways, multiple sequencers, or a combination thereof. The given compute node may be further configured to take action based on a given functionally equivalent message of the multiple functionally equivalent messages, the given functionally equivalent message being the first to arrive at the given compute node. The given compute node may be further configured to ignore other functionally equivalent messages of the multiple functionally equivalent messages that arrive after the given functionally equivalent message.

In this manner, multiple compute node-destined messages representing the same message may be received from multiple gateways 120 at a given compute node, such as compute node 140-1. This may be the case when electronic trading system 100 is configured for high availability (HA). For example, redundant flows from participant devices may be received between multiple gateways, and thus multiple compute node-destined messages representing the same message may be transmitted by the gateways to compute node 140. According to an example embodiment, a given core compute node may be configured to take action based on a message destined for the given compute node among the multiple compute node-destined messages, which may be the first to arrive at the given compute node relative to other compute node-destined messages among the multiple compute node messages representing the same message. A given core compute node may further be configured to ignore messages destined for other compute nodes that arrive after the message destined for the given compute node. Such messages destined for multiple compute nodes that represent the same message may be referred to as functionally equivalent messages. The following description of Figure 1E describes at least one example of how a recipient node may determine that two or more such messages destined for a compute node are functionally equivalent and represent the same message.

The electronic trading system 100 may be an active electronic trading system in which at least one of the plurality of sequencers may be communicatively coupled to a disaster recovery site that includes a standby electronic trading system, such as disaster recovery site 155 of FIG. 1D disclosed below.

FIG. 1D is a block diagram of another example embodiment of an electronic trading system. FIG. 1D illustrates an example electronic trading system 100 that includes multiple gateways 120-1, 120-2, ..., 120-g (collectively referred to as gateways 120), a set of core compute nodes 140-1, 140-2, ..., 140-c (collectively referred to as core compute nodes 140 or compute nodes 140), and one or more sequencers 150-1, 150-2, ..., 150-s (collectively referred to as sequencers 150). Accordingly, in some embodiments, gateways 120, core compute nodes 140, and sequencers 150 are considered nodes in electronic trading system 100. As described in more detail below, in one embodiment, gateways 120, compute nodes 140, and sequencers 150 are directly connected to one another, preferably via a low-latency, dedicated connection 180.

The term "peer" in the context of describing electronic trading system 100 refers to other devices that generally serve the same function in electronic trading system 100 (e.g., "gateway" vs. "core compute node" vs. "sequencer"). For example, gateways 120-2, ..., 120-g are peers of gateway 120-1, core compute nodes 140-2, ..., 140-c are peers of core compute node 140-1, and sequencers 150-2, ..., 150-s are peers of sequencer 150-1.

The terms "active" and "standby" in connection with describing system 100 may refer to the high availability (HA) role/state/mode of a system/component. Generally, a standby system/component is a redundant (backup) system/component that is capable of taking over the functions performed by an active system/component when powered on. Such a switchover/failover, i.e., switching from a standby role/state/mode to an active role/state/mode, may be performed automatically in response to a failure of the currently active system/component, as a non-limiting example.

The electronic trading system 100 processes trade orders and provides related information to one or more participant computing devices 130-1, 130-2, ..., 130-p (collectively, participant devices 130). The participant devices 130 interact with the electronic trading system 100 and may be one or more personal computers, tablets, smartphones, servers, or other data processing devices configured to display and receive trade order information. The participant devices 130 may be operated by humans via a graphical user interface (GUI) or via high-speed automated trading methods running on a physical or virtual data processing platform. Each participant device 130 may exchange messages with (i.e., send and receive messages from) the electronic trading system 100 via a connection established by the gateway 120. While FIG. 1D illustrates each participant device 130 as connected to the electronic trading system 100 via a single connection to the gateway 120, it should be understood that a participant device 130 may be connected to the electronic trading system 100 via multiple connections to one or more gateway devices 120.

Note that each gateway 120-1 may serve a single participant device 130, but typically serves multiple participant devices 130.

Compute nodes 140-1, 140-2, ..., 140-c (also referred to herein as matching engines 140 or compute engines 140) provide the matching functionality described above and may also generate outgoing messages that are distributed to one or more participant devices 130. Each compute node 140 is a high-performance data processor and typically maintains one or more data structures that search and maintain one or more order books 145-1, 145-2, ..., 145-b. An order book 145-1 may be maintained, for example, for each product for which core compute node 140-1 is responsible. One or more of the compute nodes 140 and/or one or more of the gateways 120 may provide a market data feed 147. The market data feed 147 may be broadcast (e.g., multicast) to participants, which may be participant devices 130 or any other suitable computing devices.

Some outgoing messages generated by core compute nodes 140 may be synchronous, i.e., generated directly by one core compute node 140 in response to one or more incoming messages received from one or more participant devices 130, such as an outgoing "approval message" or "execution message" in response to a corresponding incoming "new order" message. However, in some embodiments, at least some outgoing messages may be asynchronous and initiated by trading system 100, such as, for example, certain "unsolicited" cancellation messages and "trade stop" or "trade failure" messages.

A distributed computing environment such as the electronic trading system 100 can be configured with multiple matching engines running in parallel on multiple compute nodes 140.

The sequencer 150 ensures that the proper ordering of any order-dependent operations is maintained. To ensure that operations on incoming messages are not performed out of order, incoming messages received at one or more gateways 120, such as a new trade order message from one of the participant devices 130, typically subsequently pass through at least one sequencer 150 (e.g., a single currently active sequencer and possibly one or more standby sequencers), and the incoming message is marked with an order identifier (by the single currently active sequencer, if multiple sequencers are present). The identifier may be a unique, monotonically increasing value used during subsequent processing throughout the distributed system 100 (e.g., the electronic trading system 100) to determine relative ordering among messages and to uniquely identify messages throughout the electronic trading system 100. In some embodiments, the order identifier may be an indication of the order (i.e., sequence) in which the message arrived at the sequencer. For example, the order identifier may be a value that is monotonically incremented or decremented by the sequencer according to a fixed interval for each arriving message; e.g., the order identifier may be incremented by one for each arriving message. However, it should be understood that the order identifier, while unique, is not limited to being a monotonically increasing or decreasing value. In some embodiments, the original unmarked message and the order-marked message may be essentially identical, except for the order identifier value contained in the marked version of the message. Once ordered, the marked incoming messages, i.e., the order-marked messages, are typically then forwarded by the sequencer 150 to other downstream compute nodes 140 to potentially perform order-dependent processing on the messages. Thus, in addition to uniquely identifying messages throughout the electronic trading system 100, the order identifiers assigned by the sequencer 150 may also determine the relative ranking of each marked message among other marked messages in the electronic trading system 100.

Thus, as opposed to other purposes for which sequence identifiers may be employed, the unique sequence identifiers disclosed herein may be used to ensure a deterministic ranking (i.e., order) for electronic transaction message processing. The unique sequence identifier represents a unique, deterministic ranking (i.e., order) that is indicative of the processing of a given electronic transaction message relative to other transaction messages within an electronic transaction system. According to an example embodiment, the sequence identifier may be added to the sequence ID field 110-14 of a message, as further disclosed below with respect to FIG. 1E, by way of non-limiting example.

In some embodiments, messages may flow in the other direction, i.e., from a core compute node 140 to one or more of the participant devices 130, passing through one or more of the gateways 120. Outgoing messages generated by one such core compute node 140 may also be order-dependent (i.e., sequence-order dependent) and therefore, more typically, may first pass through a sequencer 150 that is marked with an order identifier. The sequencer 150 may then forward the marked response messages to the gateway 120 for delivery to the participant devices 130 in the appropriate deterministic order.

The use of sequencer 150 to generate and mark messages or representations thereof with unique sequence numbers, i.e., to generate sequence-marked messages, ensures that the correct ordering of operations is maintained throughout the distributed system, i.e., electronic trading system 100, regardless of which compute node or set of compute nodes 140 processes messages. This approach provides "state determinism," i.e., the entire state of the system is deterministic and reproducible (possibly elsewhere, such as a disaster recovery site), providing fault tolerance, high availability, and disaster recoverability.

It may also be important for a producing node (i.e., a node that introduces a new message into the electronic trading system 100, e.g., by generating a new message and/or forwarding a message received from a participant device 130) and its peer nodes to receive a sequence number assigned to that message. Receiving a sequence number for a message it generates may be useful for the producing node and its peer nodes not only to process the messages in order according to their sequence numbers, but also to correlate messages generated by the node with message sequence identifiers used throughout the rest of the electronic trading system 100. Such correlation between an unmarked version of a message as introduced into the electronic trading system by the producing node and an order-marked version of the same message output by the sequencer may be made via identifying information in both versions of the message, as described further below in connection with FIG. 1E. Subsequent messages generated within the electronic trading system 100 may also be assigned their own sequence numbers, but may still reference one or more sequence numbers of related, preceding messages. Thus, a node may need to quickly reference (by sequence number) messages it previously generated. This is because, for example, the sequence number of the message generated by the node is used as the reference for subsequent messages.

In some embodiments, the producing node may first send a message to the sequencer 150 and wait to receive a sequence number for the message from the sequencer before forwarding the message to other nodes in the electronic trading system 100.

In an alternative exemplary embodiment, to avoid at least one hop that may add unnecessary latency within electronic trading system 100, after receiving an unordered message from the generating node, sequencer 150 may not only send an ordered version of the message (i.e., an order-marked message) to the destination node, but may also substantially simultaneously return the ordered version of the message to the sending node and its peers. For example, after sequencer 150 assigns a sequence number to an incoming message sent from gateway 120 to core compute node 140, sequencer 150 may not only forward the ordered version of the message to core compute node 140, but may also return the ordered version of the message to gateway 120-1 and other gateways 120. Thus, any gateway 120 can easily identify the associated message originally generated by gateway 120-1 by its sequence number when any subsequent messages generated at core compute node 140 are based on that sequence number.

Similarly, in some further embodiments, an ordered version of an outgoing message generated by core compute node 140 and sent therefrom to gateway 120 and ordered by sequencer 150 may be forwarded by sequencer 150 to gateway 120 and sent back to core compute node 140.

Some embodiments may include multiple sequencers 150 for high availability, for example, to ensure that other sequencers are available if the first sequencer fails, as further disclosed below with respect to FIG. 4. For embodiments with multiple sequencers 150 (e.g., a currently active sequencer 150-1 and one or more standby sequencers 150-2, ..., 150-s), the currently active sequencer 150-1 may maintain a system status log (not shown) of all messages passed through sequencer 150-1 and the messages' associated sequence numbers. This system status log may be continuously or periodically sent to standby sequencers, providing them with the necessary system state to enable them to take over for the active sequencer if necessary. Alternatively, the system status log may be stored in a data store accessible to the multiple sequencers 150.

The system state log may be continuously or periodically replicated to one or more sequencers in a standby replica electronic trading system (not shown in detail) at disaster recovery site 155, so that in the event of a catastrophic failure of the primary site of electronic trading system 100, electronic trading can continue in exactly the same state at disaster recovery site 155.

According to an example embodiment, a currently active sequencer of the plurality of sequencers may store a system state log in a data store (not shown). The data store may be accessible to the plurality of sequencers via a shared sequencer network, such as sequencer-wide shared network 182-s, further disclosed below with respect to FIG. 1D. When a given sequencer of the plurality of sequencers switches its role (state) from standby to active, it may retrieve the system state log from the data store and synchronize its state with the state of the previously active sequencer.

In some embodiments, the system state log may be provided to a drop copy service 152, which may be implemented by one or more of the sequencers and/or by one or more other nodes in the electronic trading system 100. The drop copy service 152 may provide a record of daily trading activity throughout the electronic trading system 100, which may be distributed, for example, to regulatory authorities and/or clients, which may be connected via participant devices 130. In alternative embodiments, the drop copy service 152 may be implemented in one or more of the gateways 120. Still further, in addition to or instead of referencing the system state log, the drop copy service 152 may provide a record of trading activity based on the content of incoming and outgoing messages sent throughout the electronic trading system 100. For example, in some embodiments, the gateway 120 implementing the drop copy service 152 may receive all messages exchanged throughout the electronic trading system 100 from the sequencer 150 (and/or from the core compute nodes 140 and other gateways 120). Participant devices 130 configured to receive daily trading activity records from the drop copy service 152 do not necessarily transmit trading orders to or utilize the matching functionality of the electronic trading system 100.

Messages exchanged between participant devices 130 and gateway 120 may conform to any suitable protocol (for convenience, referred to as a "financial transaction protocol") that may be used for financial transactions. For example, messages may be exchanged according to custom protocols or established standard protocols, including both binary protocols (such as Nasdaq Ouch and NYSE UTP) and text-based protocols (such as NYSE FIX CCG). In some embodiments, electronic trading system 100 may support the simultaneous exchange of messages according to multiple financial transaction protocols, including multiple protocols, simultaneously on the same gateway 120. For example, participant devices 130-1, 130-2, and 130-3 may simultaneously establish trading connections and exchange messages with gateway 120-1 according to Nasdaq Ouch, NYSE UTP, and NYSE FIX CCG, respectively.

Furthermore, in some embodiments, the gateway 120 may translate messages conforming to a financial transaction protocol received from a participant device 130 into a standardized (e.g., standardized) message format used to exchange messages between nodes in the electronic trading system 100. The standardized transaction format may be an existing protocol and may generally be of a different size and data format than any of the financial transaction protocols used to exchange messages with the participant device 130. For example, the standardized transaction format may optionally include one or more additional fields or parameters, or omit one or more fields or parameters, when compared to the financial transaction protocol of the original incoming message received at the gateway 120 from the participant device 130, and/or each field or parameter of the message in the standardized format may be of a different data type or size than the corresponding message received at the gateway 120 from the participant device 130. Similarly, in the reverse direction, the gateway 120 may translate outgoing messages generated in a standardized format by the electronic trading system 100 into messages in the format of one or more financial transaction protocols used by the participant device 130 to communicate with the gateway 120.

FIG. 1E is a table of an example embodiment of fields of a message format 110 for a transaction message, such as a transaction message exchanged between nodes in the electronic trading system 100 disclosed above. In the example embodiment of FIG. 1E, the message format 110 is a standardized message format intended to be used for internal (i.e., within the electronic trading system 100) representation of transaction messages as they are exchanged between nodes in the electronic trading system 100. In this example embodiment, the gateway 120 exchanges messages between the participant 130 and the electronic trading system 100 and translates the messages between a format specified by one or more financial transaction protocols used by the participant 130 and the standardized transaction format used between nodes in the electronic trading system 100. It should be understood that fields 110-1 through 110-17 are for non-limiting example purposes, that the message format 110 may include more, fewer, or different fields, and that the order of such fields is not limited to that shown in FIG. 1E.

While the fields in message format 110 are shown in a single message format in this example, they may be distributed across multiple message formats or encapsulated in a layered protocol. For example, in other embodiments, some sets of fields in message format 110 may be included as part of a header, trailer, or extension field in a layered protocol that encapsulates other fields of message format 110 in the message payload. According to some example embodiments, message format 110 may define one or more fields of data encapsulated in the payload (data) portion of other message formats, including, without limitation, the respective payload portions of IP datagrams, UDP datagrams, and TCP packets, or message data frame formats such as, by way of non-limiting example, Ethernet data frame formats, or other data frame formats, including InfiniBand, Universal Serial Bus (USB), PCI Express (PCI-e), and High-Definition Multimedia Interface (HDMI).

Message format 110 includes fields 110-1...110-6 corresponding to information that may be included in messages sent or received in accordance with a financial transaction protocol for communication with one or more participant devices 130. As a non-limiting example, message type field 110-1 indicates the transaction message type. Some transaction message types (such as the message types "New Order," "Replacement Order," or "Cancel Order") correspond to messages received from a participant device 130, while other message types (such as "New Order Acknowledgement," "Replacement Order Acknowledgement," "Cancel Order Acknowledgement," "Execution," "Execution Report," "Unaccepted Cancellation," "Trade Failure," or various rejection messages) correspond to messages included in transaction messages generated by electronic trading system 100 and sent to participant devices 130.

The message format 110 also includes a stock symbol field 110-2 that contains an identifier for the security being traded, such as a stock symbol or stock ticker. For example, "IBM" is the stock symbol for "International Business Machines Corporation." The side field 110-3 in the message format 110 may be used to indicate the "side" of the trade message, such as whether the trade message is a "buy," "sell," or "short sale." Similarly, the price field 110-4 may be used to indicate the desired price at which to buy or sell the security. The quantity field 110-5 may be used to indicate the desired quantity of the security (e.g., number of shares). The message format 110 may include an order token field 110-6, which may be appended to the "order token" or "client order ID" originally provided by the participant device 130 to uniquely identify the new order within the context of a particular trading session (i.e., "connection" or "flow") established between the participant device 130 and the electronic trading system via the gateway 120.

While fields 110-1...110-6 are representative fields typically included for most message types according to most financial transaction protocols, it should be understood that message format 110 may also include additional or alternative fields to specifically support particular message types or particular financial transaction protocols. For example, according to many financial transaction protocols, the "Replacement Order" and "Cancel Order" message types require the participant 130 to provide an additional order token to represent the replaced or canceled order and distinguish it from the original order. Similarly, "Replacement Orders" and "Cancel Orders" also typically include a replacement/cancel quantity field, and "Replacement Orders" may include a replacement price field. These additional replacement/cancel order token, replacement price, and replacement/cancel quantity fields may be included in the corresponding acknowledgment message sent by electronic trading system 100.

Additionally, message format 110 includes fields 110-11...110-17 that may be used internally within electronic trading system 100 but do not necessarily correspond to fields in messages exchanged with participant devices 130. For example, node identifier field 110-11 may uniquely identify each node in electronic trading system 100. In some embodiments, a generating node may include its identifier in messages it introduces to electronic trading system 100. For example, each gateway 120 may include its node identifier in messages it forwards from participant devices 130 to compute nodes 140 and/or sequencer 150. Similarly, each compute node 140 may include its node identifier in messages it generates and sends to other nodes in electronic trading system 100 (e.g., acknowledgements, executions, or types of asynchronous messages intended for ultimate forwarding to one or more participant devices 130). Thus, each message introduced to electronic trading system 100 may be associated with the generating node of the message via node identifier field 110-11 in the message.

The message format 110 may also include a flow identifier field 110-12. In some embodiments, each trading session (i.e., "connection" or "flow") established between a participant device 130 and a gateway 120 may be identified by a flow identifier that is intended to be unique throughout the electronic trading system 100. As described above in connection with FIG. 1D, a participant device 130 may be connected to the electronic trading system 100 via one or more flows and via one or more of the gateways 120. In such embodiments, the version of the message in the standardized message format 110 (used between nodes in the electronic trading system 100) for all messages exchanged between a participant device 130 and the electronic trading system 100 via a particular flow will include a unique identifier for that flow in the flow identifier field 110-12. In some embodiments, the flow identifier field 110-12 is added by the node that generated the message. For example, the gateway 120 may populate the flow identifier field 110-12 with an identifier of a flow associated with a message that the gateway 120 introduces into the electronic trading system 100 and that it receives from a participant 130. Similarly, the core compute node 140 may populate the flow identifier field 110-12 with a flow identifier associated with a message that it generates (i.e., a response message such as an acknowledgement message or a trade, or other outgoing message, including an asynchronous message).

In some embodiments, the flow identifier field 110-12 contains a value that uniquely identifies a logical flow, which may be implemented as multiple redundant trading session connections, possibly via multiple gateways, indeed for high availability purposes. That is, in some embodiments, the same flow ID may be assigned to two or more redundant flows between a participant device 130 and a gateway 120. In such embodiments, the redundant flows may be in either an active/standby or active/active configuration. In an active/active configuration, functionally equivalent messages may be exchanged in parallel between a participant device 130 and a gateway 120 simultaneously via multiple redundant flows. That is, a trading client may send functionally equivalent messages to the electronic trading system 100 simultaneously via multiple redundant flows and receive multiple functionally equivalent responses from the electronic trading system 100 simultaneously via multiple redundant flows. However, the electronic trading system 100 may only act on a single such functionally equivalent message. In an active/standby configuration, a single flow among the multiple redundant flows may be designated as the active flow at a time, while the other flow among the multiple redundant flows may be designated as the standby flow. Transaction messages are actually only exchanged over the currently active flow. Whether the redundant flows are configured in an active/active or active/standby configuration, messages exchanged over either of the redundant flows can be identified by the same flow identifier stored by the message's originating node in the flow identifier field 110-12 of the standardized message format 110.

As described above, in some embodiments, messages exchanged between nodes in the electronic trading system 100 are sent to the sequencer 150 to be marked with a sequence identifier. Accordingly, the message format 110 includes a sequence identifier field 110-14. In some embodiments, an "unmarked message" may be sent with the sequence identifier field 110-14 set to an empty blank (e.g., zero) value. In other embodiments, the sequence identifier field 110-14 of an unmarked message may be set to a specific predetermined value that the sequencer does not assign to the message, or to an invalid value. Still other embodiments may specify that the message is not marked via an indicator in another field (not shown) of the message, such as a Boolean or flag value indicating whether the message is sequenced. When the sequencer 150 receives an unmarked message, it may subsequently generate a "sequence-marked message" by adding a valid sequence identifier value to the sequence identifier field 110-14 of the unmarked message. A valid sequence identifier value in the sequence identifier field 110-14 of a sequence-marked message uniquely identifies the message and also specifies the marked message's definitive position in the relative ordering of marked messages among other marked messages throughout the electronic trading system 100. In this example, a "sequence-marked message" sent by the sequencer 150 may be identical to a corresponding unmarked message subsequently received by the sequencer, except that the sequence identifier field 110-14 of the sequence-marked message contains a valid sequence identifier value.

In some embodiments, the message format 110 may also include a reference order identifier field 110-15. A generating node may populate the reference order identifier field 110-15 of a new message it generates with the value of the order number of a previous message related to the message being generated. The value of the reference order identifier field 110-15 allows nodes in the electronic trading system 100 to correlate the message with the previous related message.

A previously related message referenced in the reference sequence identifier field 110-15 may be a previous message in the same "order chain" (i.e., "trade order chain"). According to most financial transaction protocols, messages may be logically grouped into an "order chain," which is a set of messages through a single flow that reference or "derive from" a common message. An order chain typically begins with a "new order message" sent by a participant device 130. The next message in the order chain is typically a response by the electronic trading system (e.g., either a "new order acknowledged" message if the message is accepted by the trading system, or a "new order rejected" message if the message is rejected by the trading system, perhaps because it has invalid formatting or invalid parameters, such as, but not limited to, an invalid price). An order chain may also include a "cancel order" message sent by a participant device 130 that cancels at least a portion of the quantity of a previously acknowledged new order (but still outstanding, i.e., including at least some quantity not canceled and/or executed). The "cancel order" message will again be accepted or rejected by the electronic trading system with a "cancel order accept" or a "cancel order reject" message, which may also be part of an order chain. The order chain may also include a "replacement order" message sent from a participant device 130 that replaces the quantity and/or price of a previously accepted (but still outstanding) new order. The "replacement order" message will again be accepted or rejected by the electronic trading system with a "replacement order accept" or a "replacement order reject" message, which may also be part of an order chain. A previously accepted order that is still outstanding may be matched with one or more counter orders on the opposite side (i.e., a "buy" on one side and a "sell" or "short sell" on the other side). The electronic trading system 100 will then generate a full "fill" message (if all of the open order quantity is filled in a single match) or one or more partial "fill" messages (if only a portion of the open order quantity is filled in a single match), which may also be part of an order chain. As noted above, the reference sequence identifier may generally identify other previous messages in the same order chain.

For example, returning to the reference sequence identifier field 110-15, the value for the reference sequence number would be the sequence number assigned by the sequencer to an "incoming" message originating from a participant device 130 and introduced into the electronic trading system 100 by the gateway 120, such that a corresponding "outgoing" message, such as a response message generated by a compute node 140, may reference the sequence number value of the incoming message to which it is responding. In this example, a "new order acknowledgement" message or "fill" message generated by a compute node 140 would include in the reference sequence identifier field 110-15 a value for the sequence identifier assigned to the corresponding "new order" message to which the compute node 140 is responding with the "new order acknowledgement" message or fulfilling the order with the "fill" message. In general, however, the value for the reference sequence identifier field 110-15 need not necessarily be that of the message being directly responded to by the electronic trading system 100, but could be that of a previous message that is part of the same order chain, e.g., a "new order" or "new order acknowledgement" message.

In some embodiments, for at least some message types, gateways 120 may populate the reference sequence identifier field 110-15 in messages they introduce to electronic trading system 100 with the sequence identifier value for the associated previous message. For example, gateways 120 may populate the reference sequence identifier field 110-15 in a "Cancel Order" or "Replacement Order" message with the sequence identifier value assigned to the previous corresponding "New Order" or "New Order Acknowledge" message. Similarly, core compute node 140 may populate the sequence identifier field 110-15 for the corresponding "Cancel Order Acknowledge" or "Replacement Order Acknowledge" message with the sequence identifier value for the "New Order" or "New Order Acknowledge" message, rather than the sequence identifier for the message to which compute node 140 is directly responding (e.g., rather than the sequence identifier of the "Cancel Order" or "Replacement Order" message). Again, the reference sequence identifier field 110-15 allows nodes in electronic trading system 100 to generally correlate a message with one or more previous messages in the same order chain.

A producing node may include a node-specific timestamp field 110-13 in messages that it introduces into the electronic trading system 100. While the sequence identifier included in the sequence identifier field 110-14 of the sequence-marked message output by the sequencer 150 is intended to be unique throughout the electronic trading system 100, the value in the node-specific timestamp field 110-13 may be unique among a subset of messages introduced into the electronic trading system 100 by a particular producing node. Although referred to herein as a "timestamp," the value placed in the node-specific timestamp field 110-13 may be any suitable value that is unique among messages produced by that node. For example, the node-specific timestamp may actually be a timestamp or any suitable monotonically increasing or monotonically decreasing value.

Some embodiments may include other timestamp fields in the message format. For example, some message formats may include a reference timestamp field, which may be a timestamp value assigned by the generating node of the previous associated message. In such embodiments, a compute node 140 may include a new timestamp value in the node-specific timestamp field 110-13 for messages it generates, and may include the timestamp value from the associated message in the reference timestamp field of messages it generates. For example, a "New Order Acknowledgement" message generated by a compute node may include the timestamp value of the "New Order" to which it is responding in the reference timestamp field of the "New Order Acknowledgement Message." Furthermore, in some embodiments, compute nodes 140 may not include a new timestamp value in the node-specific timestamp field 110-13 in messages they generate, but may simply add the timestamp value from the previous associated message to that node-specific timestamp field 110-13.

The message format 110 may also include an entity type field 110-16 and an entity count field 110-17. The entity type of a message may depend on whether it is introduced into the electronic trading system 100 by the gateway 120 or by the compute node 140; in other words, whether the message is an incoming message being received at the gateway 120 from a participant device 130 or an outgoing message being generated by the compute node 140 to be sent to a participant device 130. For example, in some embodiments, incoming messages are considered to be of entity type "flow" (and a value representing type "flow" is added to the entity type field 110-16 by the gateway 120), while outgoing messages are considered to be of entity type "stock symbol" (and a value representing type "stock symbol" is added to the entity type field 110-16 by the compute node 140). In such embodiments, an entity count of type "flow" is maintained by the gateway 120, and an entity count of type "stock symbol" is maintained by the compute node 140.

Considering the entity type "flow," the gateway 120 maintains a per-flow incoming message count, counting the incoming messages received by the gateway 120 through each flow active on the gateway 120. For example, if four non-redundant flows are active on the gateway 120, each flow would be assigned a unique flow identifier as described above, and the gateway 120 would maintain a per-flow incoming message count, counting the number of incoming messages received through each of those four flows. In such an embodiment, the gateway 120 adds to the entity count field 110-17 of the incoming message the per-flow incoming message count associated with the flow of the incoming message (as identified throughout the electronic trading system 100 by the flow identifier value added to the message's flow identifier field 110-12).

In the case of redundant flows in an active/active configuration (i.e., a configuration in which multiple flows receive the same message, or at least functionally equivalent messages, in parallel from participant devices 130 connected via one or more gateways 120, as described above), each underlying redundant flow will be assigned the same flow identifier; further, the per-flow incoming message count may still be kept separate for each redundant flow, particularly if the redundant flows are implemented on separate gateways 120. Because the participant devices 130 are expected to send the same set of messages in the same order (i.e., sequence) to the electronic trading system 100 via each of the redundant flows, it is also expected that the entity counts assigned to functionally equivalent messages received via the separate redundant flows should be identical. These functionally equivalent incoming messages may be forwarded by the gateways 120 to the sequencer 150 and the compute node 140. Thus, in such embodiments, sequencer 150 and compute node 140 may receive multiple functionally equivalent incoming messages associated with the same flow identifier, but if the entity counts for the multiple messages with the same flow identifier are identical, sequencer 150 and compute node 140 may identify such messages as functionally equivalent. In some embodiments, sequencer 150 and compute node 140 may track, for each flow, the highest entity count contained in the entity count field 110-17 of the incoming messages associated with each flow. This allows sequencer 150 and compute node 140 to act only on the first of multiple functionally equivalent incoming messages received by each node and ignore other functionally equivalent incoming messages that arrive later. For example, in some embodiments, sequencer 150 may order only the first such functionally equivalent incoming message, and compute node 140 may begin processing only on the first such functionally equivalent message. If an incoming message received by a node (i.e., sequencer 150 or compute node 140) has an entity count less than or equal to the highest entity count known to the node for that flow, the node may infer that the incoming message is functionally equivalent to other previously received incoming messages and may simply ignore the subsequently received functionally equivalent incoming messages.

Considering now the case of entity type "stock symbol," compute node 140 may maintain a per-stock-symbol outgoing message count, counting the outgoing messages generated by and sent from compute node 140 for each stock symbol handled by compute node 140. For example, if four stock symbols (e.g., MSFT, GOOG, IBM, ORCL) are handled by compute node 140, each stock symbol would be assigned a stock symbol identifier added to message stock symbol field 110-2, as described above, and compute node 140 would maintain a per-stock-symbol outgoing message count, counting the number of outgoing messages it generated and sent while handling each of these four stock symbols. In such an embodiment, compute node 140 adds the per-stock-symbol outgoing message count associated with the outgoing message's stock symbol (as identified throughout electronic trading system 100 by the value added to message stock symbol field 110-2) to the incoming message's entity count field 110-17.

In some embodiments, as described further below, compute nodes may be configured such that multiple compute nodes handle a particular stock symbol in parallel for high availability reasons. Due to the deterministic ordering of messages throughout the electronic trading system 100 provided by the sequencer 150, even if multiple compute nodes are handling a given stock symbol, they will be processing incoming messages that similarly reference the same stock symbol in the same order (i.e., sequence), thereby ensuring that functionally equivalent response messages are generated in parallel. Considering outgoing messages sent across multiple compute nodes 140 for a particular stock symbol, each outgoing message referencing that stock symbol should have a functionally equivalent message sent by each other compute node 140 actively handling that stock symbol. All of these outgoing messages may be sent by the compute nodes 140 to the sequencer 150 and the gateway 120. Thus, in such an embodiment, sequencer 150 and gateway 120 may receive multiple functionally equivalent incoming messages associated with the same stock symbol, but sequencer 150 and gateway 120 may identify messages as functionally equivalent if the entity counts are identical for multiple messages with the same stock symbol identifier. In some embodiments, sequencer 150 and gateway 120 may track, for each stock symbol, the highest entity count included in the entity count field 110-17 of an outgoing message associated with that stock symbol. This allows sequencer 150 and gateway 120 to act only on the first of multiple functionally equivalent outgoing messages received by each node and ignore other functionally equivalent outgoing messages that arrive later. For example, sequencer 150, in some embodiments, may order only the first such functionally equivalent outgoing message that arrives. Similarly, gateway 120 may begin processing only the first such functionally equivalent message that arrives. If an outgoing message received by a node (i.e., sequencer 150 or gateway 120) has an entity count that is less than or equal to the highest entity count previously known to the node for that stock symbol, the node may infer that the outgoing message is functionally equivalent to other previously received outgoing messages and may simply ignore subsequently received functionally equivalent outgoing messages.

In embodiments in which the sequencer 150 orders only the first message of multiple functionally equivalent messages arriving at the sequencer, the sequencer can do so in various ways. In one example, other subsequently arriving messages that are functionally equivalent to the first arriving functionally equivalent message may simply be ignored by the sequencer (in which case only a single order-marked message for a set of functionally equivalent messages may be output by the sequencer). Another possibility is for the sequencer to track the sequence number it assigns to the first functionally equivalent message by, for example, correlating between the message's entity count, its flow identifier or stock symbol identifier (for messages with entity types of "flow" and "stock symbol," respectively), and its sequence number. The sequencer can then output an ordered version of each functionally equivalent message in which the value of the order identifier field 110-14 for all the order-marked versions of the functionally equivalent message is the same as that assigned by the sequencer to the first arriving message among the functionally equivalent messages received by the sequencer 150.

In other embodiments, sequencer 150 may not track whether messages are functionally equivalent and may assign a unique sequence number to each unordered message that arrives at sequencer 150, regardless of whether the message is one of multiple functionally equivalent messages. In such embodiments, each order-marked version of a message among multiple functionally equivalent messages is assigned a different order identifier by the sequencer as the value in order identifier field 110-14. To determine a valid order identifier for a set of functionally equivalent messages, a recipient node of an ordered, functionally equivalent message in such an embodiment may use the order identifier in the ordered version of the ordered, functionally equivalent message that arrives at the node first. In embodiments where there are direct point-to-point connections between nodes in electronic trading system 100, order-marked versions of messages are sent out in the ordered order by sequencer 150 and should be received in the same ordered order among all nodes directly connected to the sequencer. Therefore, for all nodes receiving sequence-marked messages via their respective direct point-to-point connections with the sequencer, the first-arriving sequence-marked message of multiple functionally equivalent sequence-marked messages should have the same value in sequence identifier field 110-14.

As will become apparent from the above, in addition to a message sequence identifier, in embodiments having a message format such as message format 110, there may be numerous other aspects that uniquely identify a message throughout electronic trading system 100. For example, in embodiments in which a message includes both a node identifier and a node-specific timestamp, the presence of these two identifiers in a message may be sufficient to uniquely identify the message throughout electronic trading system 100. Such fields may be understood to include metadata. Multiple messages that include such identical metadata may be understood to include common metadata. Similarly, in embodiments in which flow identifiers are unique throughout electronic trading system 100, the combination of a message's flow identifier and node-specific timestamp may be sufficient to uniquely identify a message throughout electronic trading system 100. Furthermore, the combination of a flow identifier and an entity count may be sufficient to uniquely identify a message of entity type "flow," and the combination of a stock symbol identifier and entity count may be sufficient to uniquely identify a message of entity type "stock symbol."

It should be noted, however, that while there may be other ways of uniquely identifying a message throughout the electronic trading system 100 in addition to the order identifier assigned to the message, an order identifier is still required to specify the relative ranking of the message among other messages generated by other nodes throughout the electronic trading system 100 in an unbiased and deterministic manner. For example, if the node-specific timestamp is actually implemented as a timestamp value, even if the system clocks between the nodes are perfectly synchronized, two different messages each generated by different nodes may each be assigned the same timestamp value by their respective generating nodes, and the relative ranking between these two messages will be ambiguous. Even if the messages are uniquely identifiable, the recipient nodes of both messages will still need a way to determine the relative ranking of the two messages before taking any possible action on the messages.

One possible approach for a recipient node to resolve this ambiguity is through the use of randomness, for example, by randomly selecting one message to precede another in the relative ranking of messages throughout the electronic trading system 100. However, using randomness to resolve ambiguity does not support "state determinism" throughout the electronic trading system 100. Different recipient nodes would randomly determine different relative rankings between the same set of messages, resulting in unpredictable and non-deterministic behavior within the electronic trading system 100 and hindering the proper implementation of important features such as fault tolerance, high availability, and disaster recovery.

Other approaches for recipient nodes to resolve ranking ambiguity include, for example, a predetermined prioritization scheme based on the node identifier associated with the message. However, such an approach works against the important goal of fairness by giving higher priority to some messages based solely on the node identifier of the node that introduced the message to the electronic trading system 100. For example, some participant devices 130 may be prioritized simply because they happen to be connected to the electronic trading system 100 via a gateway 120 that is considered high in the predetermined prioritization scheme.

Given that a message is uniquely identified via an entity count and either a stock symbol identifier or a flow identifier, depending on whether the message has an entity type of "stock symbol" or "flow," respectively, there can be deterministic ordering among other messages associated with that stock symbol (for messages with an entity type of "stock symbol") or that flow (for messages with an entity type of "flow"). However, ordering among other messages associated with different stock symbols and flows, respectively, remains non-deterministic.

Thus, even if other fields in message format 110 may be sufficient to uniquely identify a message throughout electronic trading system 100, a sequence identifier assigned to the message by sequencer 150 may still be necessary to fairly and deterministically specify the ranking of the message relative to other messages in electronic trading system 100. In such an embodiment, sequencer 150 (or a single currently active sequencer, if multiple sequencers 150 are present) acts as the authoritative source of truly deterministic ranking among sequence-marked messages throughout electronic trading system 100.

In some embodiments, a node in the electronic trading system 100 may receive two versions of a message: an unordered (unmarked) version of the message as introduced into the electronic trading system 100 by the originating node, and a (marked) version of the message that includes an order identifier assigned by the sequencer 150. This may occur in embodiments in which the originating node sends an unmarked message to one or more recipient nodes and the sequencer 150. The sequencer 150 may then send an order-marked version of the same message to a set of nodes that includes the same recipient node.

As noted above, while the sequence-marked version of a message is useful for determining the relative processing order (i.e., position in the sequence) of the message among other marked messages in the electronic trading system 100, it may also be useful for a recipient node to receive an unmarked version of the message. For example, it is certainly possible that an unmarked version of a message will be received before a marked version of a message in unexpected circumstances (e.g., in embodiments where there is a direct connection between nodes) because the marked version of the message is transmitted through the sequencer 150 via an intervening hop. Thus, in some embodiments, a recipient node has the opportunity to activate processing of an unmarked message in response to receiving the unmarked message, even before the recipient node receives a marked version of the message that reliably indicates the relative ordering of the marked message among other marked messages.

A node receiving both the marked and unmarked versions of the same message can correlate the two versions via the same identifying information or "common metadata" in both versions of the message. For example, as described above, a generating node may include in the message it generates (i.e., the unmarked message) a node identifier and a node-specific timestamp that together may uniquely identify each message throughout the electronic trading system 100. In embodiments in which the marked and unmarked versions of a message are essentially identical except for an order identifier assigned by the sequencer 150, the marked message will also include the same node identifier and node-specific timestamp that are also included in the corresponding unmarked message, thereby allowing recipient nodes of both versions of the message to correlate the marked and unmarked versions. Thus, while a marked message indicates the relative ranking of the marked message with respect to other marked messages throughout the electronic trading system 100, due to correlations that may occur between the unmarked and marked versions of the same message, the marked message also indicates (at least indirectly via the correlations described above) the relative ranking of that message with respect to other messages (whether marked or unmarked) throughout the electronic trading system 100. It should be understood that nodes in the electronic trading system 100 may correlate sequence-marked versions of messages with unmarked versions by other ways of uniquely identifying messages as described above. For example, correlation between sequence-marked and unmarked messages may be performed by a combination of a flow identifier and a node-specific timestamp. Such correlation may additionally or alternatively be performed by the entity count of a message along with the stock symbol identifier or flow identifier in the message for messages having entity types "stock symbol" and "flow," respectively.

In an era of high-speed trading where microseconds or even nanoseconds matter, participant devices 130 exchanging messages with electronic trading system 100 are often highly latency-sensitive, and low, predictable latency is desirable. The configuration shown in FIG. 1D addresses this requirement by providing a point-to-point mesh 172 architecture between at least each of the gateways 120 and each of the compute nodes 140. In some embodiments, each gateway 120 in the mesh 172 may have a dedicated, high-speed, direct connection 180 to the compute nodes 140 and the sequencer 150.

For example, a dedicated connection 180-1-1 is provided between gateway 120-1 (i.e., GW1) and core compute node 140-1 (i.e., Core1), a dedicated connection 180-1-2 is provided between gateway 120-1 (i.e., GW1) and core compute node 140-2 (i.e., Core2), and so on. An exemplary connection 180-g-c is provided between gateway 120-g and compute node 140-c, an exemplary connection 180-s-c is provided between sequencer 150 and core compute node 140-c (i.e., Core c), an exemplary connection 180-gw1-s1 is provided between gateway 120-1 (i.e., GW g) and sequencer 150-1, and an exemplary connection 180-c1-s1 is provided between core compute node 140-1 (i.e., Core 1) and sequencer 150-1.

It should be understood that, in some embodiments, each dedicated connection 180 in the point-to-point mesh 172 is a point-to-point direct connection that does not utilize a shared switch. A dedicated or direct connection may be referred to herein interchangeably as a direct or dedicated "link," which is a direct connection between two endpoints that is dedicated (i.e., unshared) for communication therebetween. Such a dedicated/direct link may be any suitable interconnect or interface, as further disclosed below, and is not limited to a network link, such as a wired Ethernet network connection or other type of wired or wireless network link. A dedicated/direct connection/link may also be referred to herein as an end-to-end path between two endpoints. Such an end-to-end path may be a single connection/link or may include serial connections/links. However, the bandwidth of the dedicated/direct connection/link in its entirety, i.e., from one endpoint to the other, is unshared, and neither the bandwidth nor the latency of the dedicated/direct connection/link may be affected by the resource utilization of elements traversed. For example, a dedicated/direct connection/link may traverse one or more buffers or other elements whose usage does not affect bandwidth or latency. However, a dedicated/direct connection/link does not traverse a shared network switch, because such a switch may affect bandwidth and/or latency due to its shared usage.

For example, in some embodiments, the dedicated connections 180 in the point-to-point mesh 172 may be provided in a number of ways, such as 10 Gigabit Ethernet (GigE), 25 GigE, 40 GigE, 100 GigE, InfiniBand, Peripheral Component Interconnect-Express (PCIe), RapidIO, Small Computer System Interface (SCSI), FireWire, Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or a custom serial or parallel bus. Thus, while compute engine 140, gateway 120, sequencer 150, and other components are sometimes referred to herein as "nodes," other types of interconnections or interfaces are possible, the use of terms such as "compute node," "gateway node," "sequencer node," or "mesh node" should not be interpreted to imply that particular components are necessarily connected using a network link. Furthermore, a "node" as disclosed herein may be any suitable hardware, software, firmware, or combination thereof configured to perform the respective functions described above for the node. As described in more detail below, a node may be a programmed general-purpose processor, but may also be a dedicated hardware device such as a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or other hardware device or group of devices, logic within a hardware device, a printed circuit board (PCB), or other hardware component.

It should be understood that the nodes disclosed herein may be separate entities or may be integrated together within a single entity, such as within a single FPGA, ASIC, or other entity configured to execute logic to perform the functions of the node as described herein. A node may also be an entity of software-implemented logic executed by a general-purpose computer and/or any of the above devices.

Traditional approaches that connect components such as compute engines 140, gateways 120, and sequencers 150 through one or more shared switches do not provide the lowest possible latency. These traditional approaches also result in unpredictable spikes in latency during periods of increased message traffic.

In the illustrated embodiment, dedicated connections 180 are provided directly between each gateway 120 and each sequencer 150, and also between each sequencer 150 and each core compute node 140. Furthermore, in some embodiments, dedicated connections 180 are provided between all sequencers, such that the illustrated sequencer 150-1 has a dedicated connection 180 to each other sequencer 150-2, ..., 150-s. Although not shown in FIG. 1D , in some embodiments, dedicated connections 180 may also be provided between all gateways 120, such that each gateway 120-1 has a dedicated connection 180 to each other gateway 120-2, ..., 120-g, as further disclosed below with respect to FIG. 3. Similarly, in some embodiments, dedicated connections 180 are also provided between all of the compute nodes 140, such that the exemplary core compute node 140-1 has a dedicated connection 180 to each of the other core compute nodes 140-2, ..., 140-c, as further disclosed below with respect to FIG. 3.

It should also be understood that a dedicated connection 180 between two nodes (e.g., between any two nodes 120, 150, or 140) may, in some embodiments, be implemented as multiple redundant dedicated connections between those same two nodes for redundancy and increased reliability. For example, dedicated connection 180-1-1 between gateway 120-1 and core compute node 140-1 may actually be implemented as a pair of dedicated connections.

Furthermore, in some embodiments, any message sent by a node is sent in parallel to all nodes directly connected to it in point-to-point mesh 172. Each node in point-to-point mesh 172 may decide for itself, based on, for example, the node's configuration, whether to take some action in response to receiving a message or simply ignore the message. In some embodiments, a node may not completely ignore a message. Even if a node, due to its configuration, does not take any substantive action in response to receiving a message, it may still take at least some minimal action, such as consuming any sequence number assigned to the message by sequencer 150. That is, in such embodiments, the node may track the last received sequence number to ensure that if the node takes more substantive action on the message, it does so in the appropriate ordering order.

For example, suppose a message containing a trade order to "sell 10 shares of Microsoft at $190.00" originates from participant device 130-1, such as a trader's personal computer, and arrives at gateway 120-1 (i.e., GW1). The message is transmitted to all core compute nodes 140-1, 140-2, ..., 140-c, even though only core compute node 140-2 is currently performing matching on the order for Microsoft stock. All other core compute nodes 140-1, 140-3, ..., 140-c may ignore the message upon receipt or may take only minimal action on the message. For example, the only action taken by 140-1, 140-3, ..., 140-c may be to consume the sequence number assigned to the message by sequencer 150-1. The message is also sent to all of sequencers 150-1, 150-2, ..., 150-s, even though a single sequencer (in this example, sequencer 150-1) is the currently active sequencer serving the mesh. The other sequencers 150-2, ..., 150-s also receive the message, giving them the opportunity to take over as the currently active sequencer if sequencer 150-1 (the currently active sequencer) fails or if moving to a different active sequencer would increase the overall reliability of electronic trading system 100. One or more of the other sequencers (e.g., sequencer 150-2) are also responsible for relaying the system state to disaster recovery site 155. Disaster recovery site 155 may include a replica of electronic trading system 100 at another physical location, with the replica comprising physical or virtual instances of some or all of the individual components of electronic trading system 100.

By sending each message in parallel to all directly connected nodes, electronic trading system 100 reduces complexity and promotes redundancy and high availability. While all directly connected nodes receive all messages by default, multiple nodes may be configured to act in a redundant manner on the same message. Returning to the example of the "sell 10 shares of Microsoft at $190.00" order above, in some embodiments, multiple core compute nodes 140 may simultaneously match the Microsoft stock order. For example, both core compute node 140-1 and core compute node 140-2 would simultaneously match the Microsoft stock message, and after receiving the incoming "sell" order message, each of core compute node 140-1 and core compute node 140-2 may independently generate a response message, such as an acknowledgement or execution message, that is sent to gateway 120 via sequencer 150 and passed to one or more participant devices 130.

Because of the strict ordering and state determinism guaranteed by sequencer 150, each associated response message independently generated by and transmitted from core compute nodes 140-1 and 140-2 can be guaranteed to be substantially and functionally equivalent. Therefore, the architecture of electronic trading system 100 readily supports redundant processing of messages, thereby increasing system availability and resilience. In such an embodiment, gateway 120 may receive multiple associated outgoing messages from core compute node 140 for the same corresponding incoming message. Because these multiple associated response messages can be guaranteed to be equivalent, gateway 120 need only process the first received outgoing message and ignore subsequent associated outgoing messages corresponding to the same incoming message. In some embodiments, the "first" and "subsequent" messages may be identified by their associated sequence numbers, as the messages may be sequence-marked messages. However, in other embodiments, such as those in which the sequencer 150 assigns a single ordering identifier among multiple functionally equivalent messages, messages may be identified as functionally equivalent based on other identifying information in the messages, such as the values in the entity type field 110-16 and entity count field 110-17, as further described above in connection with FIG. 1E.

Thus, allowing the gateway 120 to act on the first of multiple functionally equivalent related response messages to arrive can also improve the overall latency of the electronic trading system 100. Furthermore, the electronic trading system 100 can be easily configured so that any incoming message is processed by multiple compute nodes 140, each of which generates an equivalent response message that can be processed by the gateway 120 on a first-come, first-served basis. Such an architecture provides for high availability when a compute node 140 is not handling incoming messages for a predetermined period of time (whether due to a system failure, node reconfiguration, or maintenance activity) without any perceptible impact on latency.

The architecture of the point-to-point mesh 172 of such an electronic trading system 100 not only maintains low, predictable latency and redundant processing of messages, but also provides for multiple built-in redundant paths. As can be seen, multiple paths exist between any gateway 120 and any compute node 140. If the direct connection 180-1-1 between gateway 120-1 and compute node 140-1 becomes unavailable, communication is still possible between these two elements via an alternative path, such as by traversing one of the sequencers 150 instead. Thus, more generally, in the point-to-point mesh 172, multiple paths exist between any node and any other node.

Furthermore, this point-to-point mesh architecture inherently maintains another important goal of financial trading systems: fairness. A point-to-point architecture with direct connections between nodes ensures that the paths between any gateway 120 and any core compute node 140, or between the sequencer 150 and any other node, have identical or at least very similar latency. Thus, two incoming messages sent to the sequencer 150 simultaneously from two different gateways 120 should arrive at the sequencer 150 at substantially the same time. Similarly, an outgoing message sent from a core compute node 140 should be sent to all gateways 120 simultaneously and received by each gateway at substantially the same time. Because the point-to-point mesh topology does not favor any one gateway 120, the opportunity for connecting to a particular gateway 120 to unfairly impact a participant device 130 is minimized.

Furthermore, the point-to-point mesh architecture of the electronic trading system 100 allows for easy reconfiguration of a node's function, i.e., whether the node is currently acting as a gateway 120, a core compute node 140, or a sequencer 150. Performing such reconfiguration is particularly easy in embodiments in which each node has a direct connection between itself and each other node in the point-to-point mesh. When each node is connected via a direct connection to each other node in the mesh, changing a node's function within the mesh (e.g., changing a node's function from a core compute node 140 to a gateway 120, or from a gateway 120 to a sequencer 150) requires rewiring or recabling the connections 180 (whether physical or virtual) within the point-to-point mesh 172. In such embodiments, any necessary reconfiguration internal to the point-to-point mesh 172 can be easily achieved through configuration changes performed remotely. If a node is reconfigured to act as a new gateway 120, or from acting as a gateway 120 to some other function, some accompanying networking changes may be required external to the point-to-point mesh 172, but the internal wiring of the mesh may remain the same.

Thus, in some embodiments, reconfiguration of node functionality may be achieved live, even dynamically, during trading hours. For example, due to changing load characteristics or new demands on the electronic trading system 100, it may be useful to reconfigure core compute node 140-1 to act instead as an additional gateway 120. After some possible redistribution of state or configuration to other compute nodes 140, the new gateway 120 will become available and begin accepting new connections from participant devices 130.

In some embodiments, slower, potentially higher latency shared connections 182 may be provided between system components, such as between gateways 120 and/or core compute nodes 140. These shared connections 182 may be used for maintenance, control, administrative, and/or similar activities that do not require very low latency communications, as opposed to messages related to trading activity performed over dedicated connections 180 in point-to-point mesh 172, such as messages 106 and responses 107 disclosed above with respect to FIG. 1B-2. In contrast to first direct connection 180-a, second direct connection 180-b, and third direct connection 180-c, which carry traffic related to trading activity, shared connections 182g and 182c carry non-trading activity types of traffic. Shared connections 182 carrying non-trading traffic may be via one or more shared networks and via one or more network switches, and the nodes in the mesh may be distributed across these shared networks in different manners. For example, in some embodiments, the gateways 120 may all be in gateway-wide shared network 182-g, the compute nodes 140 may be in their own respective compute node-wide shared network 182-c, and the sequencers 150 may be in their own separate sequencer-wide shared network 182-s, while in other embodiments, all nodes in the mesh may communicate over the same shared network for their latency-insensitive operations.

Distributed computing environments such as electronic trading system 100 may rely on high-resolution clocks to maintain close synchronization among various components. To this end, one or more of nodes 120, 140, 150 may, in some embodiments, be provided with access to a clock, such as a high-resolution Global Positioning System (GPS) clock 195. For purposes of the following description, gateways 120, compute nodes 140, and sequencers 150 connected in point-to-point mesh 172 are referred to as "mesh nodes," which may have an architecture as further disclosed below with respect to FIG. 2.

FIG. 1F is a flow diagram of an example embodiment of a process 125 that may be performed by electronic trading system 100 to process a message. Referring to FIG. 1B-2, gateway 120-1 may receive a message from a participant device (131) and, in response, transmit message 106 to sequencer 150-1 and compute node 140-1 (132). Gateway 120-1 may optionally include an identifier generated by gateway 120-1 in message 106; in an electronic trading system with multiple compute nodes and/or sequencers, gateway 120-1 may transmit message 106 to all such nodes. In response to receiving message 106, compute node 140-1 may first determine whether message 106 is one that compute node 140-1 has assigned for processing, and may make this determination based on one or more values or identifiers in message 106 (e.g., the value of a common parameter such as a stock symbol representing a given financial instrument or a transaction type). If message 106 is assigned for processing, compute node 140-1 may perform pre-processing (134) on message 106 to generate preliminary results. For example, compute node 140-1 may load information related to values referenced in message 106 (e.g., stock symbols for financial instruments) into memory. Alternatively or additionally, in some embodiments, pre-processing may include performing matching functions, such as generating response messages, including, by way of non-limiting example, acknowledgement messages or execution messages, or performing preliminary order book updates for values referenced in message 106 (e.g., stock symbols for financial instruments). Such pre-processing may be performed such that it has no adverse side effects, or, if desired, may be "rolled back" atomically (i.e., transactionally) if it is later determined that the messages on which pre-processing was performed were received out of order.

Concurrently, sequencer 150-1 may order message 106 by, for example, associating message 106 with a unique order identifier (133) that enables message 106's position to be identified within a sequence of messages that may need to be processed in a given order (i.e., order), to generate an order-marked version of message 106, i.e., order-marked message 106'. Upon completion, sequencer 150-1 may send order-marked message 106', including message 106 (or a representation thereof) and the order identifier, to compute node 140-1 (135). When compute node 140-1 receives order-marked message 106', it may, in some embodiments, determine whether the messages 106 on which it performed its preprocessing (unordered) were received in order, i.e., verify the order (136). As part of verifying the order, compute node 140-1 may correlate message 106 with order-marked message 106' via common metadata or common identifying information in both versions of the message. In the example embodiment, in process 125, message 106 was received in order, and compute node 140-1 may proceed to continue processing message 106 (137). As a result of the pre-processing, compute node 140-1 has already completed some operations (e.g., cache lookups for values) that it would have performed after receiving order-marked message 106', thereby allowing compute node 140-1 to complete processing message 106 more quickly than it would have done absent the pre-processing operations.

FIG. 1G is a flow diagram of an example embodiment of process 165 illustrating another example embodiment of the operation of an electronic trading system. Process 165 may be executed by electronic trading system 100 to process a message. In the embodiment of FIG. 1G, electronic trading system 100 includes at least a second gateway, gateway 120-2. Referring to FIG. 1B-2, gateway 120-1 may receive a first message (M1) from a participant device (166) and, in response, transmit a message 106 corresponding to first message M1 to sequencer 150-1 and compute node 140-1 (167). While message 106 may also be referred to as first message M1, it should be understood that message 106 may not be identical to M1 because the gateway may have modified the message to include different metadata, such as, by way of non-limiting example, an identifier for gateway 120-1. Gateway 120-2 may receive a second message (M2) from another participant device (168) and may transmit a message corresponding to second message M2 to sequencer 150-1 and compute node 140-1 (169). The message corresponding to second message M2 may also be referred to herein as second message M2. In this example, the first and second messages belong to a common series of messages, such that the relative order (i.e., sequence) in which the first and second messages are processed may affect the state of electronic trading system 100. In the example embodiment of FIG. 1G, compute node 140-1 receives the second message before receiving the first message, and upon verifying that the second message is the one that compute node 140-1 assigned for processing, compute node 140-1 may then perform pre-processing on the message (171). Upon receiving the first message (M1), compute node 140-1 may also perform pre-processing on the first message (M1) (173).

Sequencer 150-1 may receive both the first and second messages. Sequencer 150-1 may then assign each of the first and second messages a respective unique sequence identifier that can be used to determine the relative ranking, i.e., order, of the first and second messages (170, 174). In some embodiments, the sequence identifiers assigned to messages by the sequencer may be, as a non-limiting example, monotonically increasing sequence numbers. Furthermore, in some embodiments, the value of the sequence identifier assigned to messages by the sequencer may be based on the arrival time of the messages at sequencer 150-1. That is, in some embodiments, as a non-limiting example, messages are placed in a first-in, first-out (FIFO) queue as they are received by sequencer 150-1 and are assigned a sequence identifier as they are removed from the FIFO queue. 1G , because message M1 was received before message M2 at sequencer 150-1, in an embodiment in which the order identifiers are monotonically increasing order numbers, sequencer 150-1 will assign message M1 a lower order number relative to the other order number assigned to message M2. After assigning order identifiers to each message is complete, sequencer 150-1 may send marked messages (i.e., messages including order identifiers) corresponding to the first and second messages or representations thereof to compute node 140-1 (175, 177). In some embodiments, once compute node 140-1 receives the marked messages, it may determine whether the (unordered) messages 106 on which it performed preprocessing were received in order, i.e., verify the order (176). As part of verifying the order, compute node 140-1 may correlate unordered message 106 with ordered marked message 106′ via common metadata or common identifying information in both versions of the messages. In operation 165, compute node 140-1 receives the second message out of order. As a result, some or all of the artifacts of prior processing (e.g., as a non-limiting example, cache lookups for values of common parameters of the messages) may optionally be discarded, or, for other types of prior processing, may be atomically rolled back. Compute node 140-1 may then proceed to process (178) both the first and second messages in the order (i.e., sequential) indicated by sequencer 150-1.

Even if compute node 140-1 determines that an unordered message was received out of order, node 140-1 still benefits from preprocessing the message. By preprocessing the message, compute node 140-1 has determined that it is likely that it will need to process this message in the future, allowing it to take preemptive action that saves time. For example, information needed to fully process the message, such as information about currently open trading orders for the same stock symbol, must be present in faster memory (e.g., FPGA memory, such as fixed logic memory 250 shown in FIG. 2, or other cache memory) before the message can be fully processed; if not already present in faster memory, it may first need to be retrieved from slower memory (e.g., a hard disk or DRAM, such as DRAM 280 shown in FIG. 2). For values of common parameters, such as stock symbols, for messages that have not been referenced recently, this information may not currently reside in fast-side memory, but compute node 140-1 may perform a memory copy operation in advance even for out-of-order messages, thereby saving time if an ordered message arrives quickly thereafter in the proper order (i.e., sequence). Provided that fast-side memory is large enough to hold information for multiple values, even receiving a sequence of multiple consecutive out-of-order messages may be beneficial by allowing the compute node to prefetch information for many values in advance. Thus, rather than ignoring the results of pre-processing operations, compute node 140-1 may temporarily store the results for reference during later processing operations.

In some example embodiments, the results of preprocessing by a compute node may remain valid in instances where the order ranking is not followed. For example, in embodiments in which each computing node handles a subset of operations, such as a subset of values for a common parameter of messages (e.g., a limited number of stock symbols or other indicators of financial instruments), a computing node may still receive messages that reference other stock symbols that it does not handle, but the order ranking may be relevant to the compute node only insofar as it affects the values that the computing node handles. Also, the ordering of messages that reference different values may be considered independent of one another. For example, if a message is received out of order relative to another message that references a stock symbol with a different value, the order in which these two messages are processed does not affect the outcome of the matching function operation, no further action (e.g., rollback) related to preprocessing is required, and the preprocessed results may ultimately be used without modification and/or bound to the state of the distributed system.

It should be understood that in some embodiments, verifying the order (136 in FIG. 1F and 176 in FIG. 1G) may not be necessary depending on the nature of the pre-processing performed by compute node 140-1. For example, if the pre-processing performed on the unordered message does not have adverse side effects and does not affect the results of processing other messages (e.g., loading data related to stock symbols referenced in the message into high-speed memory), then it may not be necessary to verify that the pre-processing of the unordered message was performed in accordance with the order determined by sequencer 150-1, and certainly it may not be necessary to discard or atomically roll back the results of the pre-processing. In such an embodiment, compute node 140-1 may simply complete processing of the message in accordance with receiving the order-marked message 106' in the order specified by the order identifier in the order-marked message 106', whenever thereafter utilizing the pre-processed results.

The amount of pre-processing that can be performed on an unordered message, and whether the results of that pre-processing need to be discarded or rolled back, may depend on fields in the message, such as the message type field 110-1, stock symbol field 110-2, side field 110-3, or price field 110-4 in the embodiment of FIG. 1E. It may also depend on whether other unordered messages that reference the same value for a common parameter in the message, such as the same stock symbol, are currently outstanding (i.e., for which a corresponding order-marked message has not yet been received).

For example, when an unordered message having a message type of "New Order" is received by core compute node 140-1, core compute node 140-1 will load stock symbol information associated with the relevant portion of the order book into high-speed memory, and if the new order is a match for an open order in the order book, compute node 140-1 will begin generating a "fill" message accordingly, but will refrain from committing to updating the order book and sending the "fill" message until it receives an ordered version of the message. However, if compute node 140-1 is also receiving other outstanding unordered "New Order" messages referencing the same stock symbol, side, price, etc. that are also potential matches for the same open order in the order book, core compute node 140-1 may perform its pre-processing differently. In some embodiments, core compute node 140-1 may generate a competing potential "fill" message for each of two outstanding unordered "new order" messages that could potentially act as a match for an outstanding order. Based on the ordered version of the message, one of the potential "fill" messages may be discarded, while the other will be committed to the order book and sent to gateway 120. In other embodiments, if two or more potential outstanding unordered messages could potentially match the same outstanding order, compute node 140-1 may not perform any pre-processing that needs to be discarded or rolled back (e.g., not create any potential "fill" messages), or may abort or suspend any such pre-processing for these outstanding unordered messages.

As another example, an outstanding unordered "New Order" message that is a potential match for an open order in the order book may conflict with an outstanding unordered "Replace Order" message or "Cancel Order" message that attempts to replace or cancel, respectively, the same open order in the order book that would otherwise serve as a potential match for the "New Order" message. In this case, depending on the relative order assigned by the sequencer to the "New Order" message versus the "Replace/Cancel Order" message, the end result may be either a match between the open order in the order book and the "New Order" message, or the open order may be canceled or replaced by a new order at a different price or quantity. Until an order-marked version of the conflicting outstanding unordered message is received by sequencer 150-1, compute node 140-1 cannot determine which of these two outcomes should occur.

In this case, compute node 140-1 may perform pre-processing differently. In some embodiments, if there are multiple conflicting unordered messages outstanding, compute node 140-1 may simply perform pre-processing that does not need to be rolled back or discarded, such as loading into fast-side memory the relevant portions of the order book associated with the stock symbols referenced in both conflicting messages. In other embodiments, compute node 140-1 may perform additional pre-processing, such as constructing one or more provisional potential responses, each corresponding to one of multiple conflict scenarios. For example, compute node 140-1 may create potential "fill" messages and/or potential "replace acknowledge" or "cancel acknowledge" messages, and possibly perform provisional updates to the order book corresponding to one or more of multiple possible outcomes. In some embodiments, compute node 140-1 may perform this additional pre-processing for all such conflict scenarios, while in other embodiments, compute node 140-1 may perform the additional pre-processing for only one or a subset of the conflict scenarios. For example, compute node 140-1 may perform additional pre-processing on unprocessed unordered messages only if there are no other outstanding conflicting unordered messages. Alternatively or additionally, compute node 140-1 may prioritize performing additional pre-processing on unprocessed conflicting unordered messages according to the amount of time and/or complexity associated with rolling back or discarding the results of the pre-processing. Upon receiving an order-marked version of an unprocessed unordered message, compute node 140-1 may then determine the order (as assigned by sequencer 150-1) in which the unprocessed unordered messages should be processed and complete processing of the messages in that order. This may, in some embodiments, include rolling back or discarding one or more results of the pre-processing.

In addition to the types of pre-processing already described above, in some embodiments, compute node 140-1 may additionally or alternatively perform pre-processing on the message regarding its validity to determine whether to accept or reject it. For example, pre-processing may include performing real-time risk checks on the message, such as verifying that the price or quantity specified by the message does not exceed a maximum value (i.e., a "max price check" or a "max quantity check"), that the stock symbol in the message is a known stock symbol (i.e., an "unknown stock symbol check"), that trading is currently permitted for that stock symbol (i.e., a "stock symbol stop check"), or that the price is properly specified with the correct number of decimal points (i.e., a "sub-penny check"). In some embodiments, a type of pre-processing may include an "anti-self-trading" validation check to prevent a particular potential match from being a self-trading, i.e., a trading client from matching against itself, if "anti-self-trading" is enabled for the particular client or trade order. If the trade order fails one or more of these validation checks, electronic trading system 100 may respond with an appropriate rejection message. While these validation checks are described in the above embodiment as being performed by compute node 140-1, it should be understood that in some embodiments, at least some of these types of validation checks may alternatively or additionally be performed by gateway 120 or other nodes in electronic trading system 100.

In further embodiments, it may be beneficial or necessary for gateway 120-1 to be informed of a unique system-wide sequence identifier associated with a client-originated message. This information may enable gateway 120-1 to match the original incoming message to a unique sequence number, which is used to ensure proper ordering of messages throughout electronic trading system 100. Such a configuration at the gateway may be required for electronic trading system 100 to achieve state determinism to provide fault tolerance, high availability, and disaster recovery with respect to activity at the gateway. One solution for configuring gateway 120-1 to maintain information about sequence identifiers associated with incoming messages is to have gateway 120-1 wait for a reply from sequencer 150-1 with the sequence identifier before forwarding the message to compute node 140-1. Such an approach may add latency to the processing of messages. In a further example, in addition to forwarding to compute node 140-1 the sequence-marked message it originally received from gateway 120-1, sequencer 150-1 may send sequence-marked messages (e.g., marked messages 106' as shown in FIG. 1B-2) in parallel to gateway 120-1. As a result, gateway 120-1 may retain sequence identifier information while minimizing latency in electronic trading system 100.

FIG. 2 is a block diagram of an example embodiment of a mesh node in a point-to-point mesh architecture of an electronic trading system, such as the electronic trading system 100 disclosed above. FIG. 2 illustrates an example embodiment of a mesh node 200 in the point-to-point mesh 172 architecture of the electronic trading system 100. The mesh node 200 may represent, for example, a gateway 120, a sequencer 150, or a core compute node 140. While the functionality in the mesh node 200 is distributed across both hardware and software in this example, the mesh node 200 may be implemented in any suitable combination of hardware and software, including pure hardware and pure software implementations, and in some embodiments, any or all of the gateway 120, the compute node 140, and/or the sequencer 150 may be implemented with commercially available components.

In the embodiment shown in FIG. 2, to achieve low latency, some functions are implemented in hardware in a fixed logic device 230, while other functions are implemented in software in a device driver 220 and a mesh software application 210. The fixed logic device 230 may be implemented in any suitable manner, including an application-specific integrated circuit (ASIC), an embedded processor, or a field-programmable gate array (FPGA). The mesh software application 210 and the device driver 220 may be implemented as instructions executing on one or more programmable data processors, such as a central processing unit (CPU). Different versions or configurations of the mesh software application 210 may be installed on a mesh node 200 depending on its role. For example, different versions or configurations of the mesh software application 210 may be installed based on whether the mesh node 200 is operating as a gateway 120, a sequencer 150, or a core compute node 140.

While any suitable physical communication link layer may be employed (including Universal Serial Bus (USB), Peripheral Component Interconnect (PCI) Express (PCI-Express, i.e., PCI-E), High-Definition Multimedia Interface (HDMI), 10 Gigabit Ethernet (GigE), 40 GigE, 100 GigE, or InfiniBand (IB), over fiber or copper cable), in this example, mesh node 200 has multiple low-latency 10 Gigabit Ethernet Small Form Factor Pluggable Plus (SFP+) connectors (interfaces) 270-1, 270-2, 270-3, ..., 270-n (collectively referred to as connectors 270). Connectors 270 may be directly connected to other nodes in a point-to-point mesh via, for example, dedicated connections 180, connected via shared connections 182, and/or connected to participant devices 130 via gateway 120. These connectors 270 are, in this example, electronically coupled to respective 10GigE media access (MAC) cores 260-1, 260-2, 260-3, ..., 260-n (collectively referred to as GigE cores 260), which in this embodiment are implemented by fixed logic device 230 to ensure minimal latency. In other embodiments, 10GigE MAC cores 260 may be implemented by a function external to fixed logic device 230, for example, in a PCI-E network interface card adapter.

In some embodiments, fixed logic device 230 may also include other components. In the example of FIG. 2, fixed logic device 230 also includes a fixed logic 240 component. In some embodiments, fixed logic component 240 may perform different functions depending on the role of mesh node 200, for example, whether it is gateway 120, sequencer 150, or core compute node 140. Fixed logic device 230 also includes fixed logic memory 250, which may be memory accessed with minimal latency by fixed logic 240. Fixed logic device 230 also includes PCI-E core 235, which may perform PCI Express functions. In this example, PCI Express is used as a conduit mechanism by device driver 220 to transfer data between hardware and software, more specifically, between fixed logic device 240 and mesh software application 210 via PCI Express bus 233. However, any suitable data transfer mechanism may be employed between hardware and software, including direct memory access (DMA), shared memory buffers, or memory mapping.

In some embodiments, mesh node 200 may also include other hardware components. For example, depending on its role in electronic trading system 100, mesh node 200 may, in some embodiments, include high-resolution clock 195 (also shown and disclosed in connection with FIG. 1D ) used to implement high-resolution clock synchronization between nodes in electronic trading system 100. Dynamic random access memory (DRAM) 280 may also be included in mesh node 200 as additional memory in connection with fixed logic memory 250. DRAM 280 may be any suitable volatile or non-volatile memory, including one or more random access memory banks, hard disks, and solid-state disks, and may be accessed via any suitable memory or storage interface. As disclosed above, mesh node 200 may represent gateway 120, sequencer 150, or core compute node 140, and may be configured in a point-to-point mesh architecture as disclosed above with respect to FIGS. 1A-D and further below with respect to FIG. 3.

1B-1, 1B-2, 1C, 1D, 1E, and 2, it should be apparent that the point-to-point mesh architecture 172 of the electronic trading system 100 provides for latency improvements in a number of ways. In embodiments in which messages are exchanged between nodes in the electronic trading system 100 via direct, dedicated connections 180 without traversing a switch, the avoidance of the switch itself enables a number of latency-related advantages.
a) The time required for a switch to process a single message (the time interval between when a message enters the switch and when it leaves the switch) is completely eliminated. This switch processing latency is typically around 1.0 microseconds, and includes buffering time and the time to route the message to its appropriate destination, as specified in the message's header.
b) Two transmission times (transmission time between the sending node and the switch and transmission time between the switch and the receiving node) are replaced by a single transmission time for transmitting a message directly between the sending node and the receiving node.
c) Communicating via direct, switch-less, point-to-point connections also eliminates the requirement to exchange messages within the mesh according to specific established protocols, such as TCP/IP, which may be required by switches but may also add processing time overhead to the sending and receiving nodes. The time required for the sending node to perform protocol-specific processing, such as constructing a TCP/IP header and calculating a checksum, and the time required for the receiving node to perform similar protocol-specific processing, such as interpreting the TCP/IP header and verifying the checksum, amounts to approximately 0.5 microseconds for each sending and receiving node, for a total of approximately 1.0 microseconds. This protocol-specific overhead may optionally be eliminated in some embodiments of point-to-point mesh architecture 172 that exchange messages according to one or more custom protocols rather than the specific established protocols required by switches, such as TCP/IP.
d) Messages may be broadcast to directly connected nodes at exactly the same time and received by directly connected recipient nodes at exactly the same time, as described further below. Different messages may also be received at exactly the same time from multiple directly connected sender nodes, as described further below.

Improved latency may be achieved in embodiments in which direct, dedicated connections 180 between nodes are implemented via dedicated communication logic and dedicated interfaces, such as GigE MAC cores 260 and connectors 270, respectively, as described above in connection with FIG. 2. In some such embodiments, each GigE MAC core 260 handles message communication between that node and a single other node, and each connector 270 is connected to a single other node in the point-to-point mesh architecture 172 via a dedicated connection 180. For example, considering the three nodes in the point-to-point mesh of the embodiment of FIG. 1D (gateway 120-1, core compute node 140-1, and core compute node 140-2), gateway 120-1 is connected to core compute node 140-1 via dedicated connection 180-1-1, and gateway 120-1 is also connected to core compute node 140-2 via a separate, dedicated connection 180-1-2. 2, GigE MAC core 260-1 and connector 270-1 for gateway 120-1 may be used solely for communications with core compute node 140-1, and GigE MAC core 260-1 and connector 270-1 for core compute node 140-1 may be used solely for communications with gateway 120-1, via dedicated connection 180-1-1. Similarly, GigE MAC core 260-2 and connector 270-2 for gateway 120-1 may be used solely for communications with core compute node 140-2, and GigE MAC core 260-1 and connector 270-1 for core compute node 140-2 may be used solely for communications with gateway 120-1, via dedicated connection 180-1-2. Thus, according to such an embodiment, dedicated compute resources, such as one of the GigE MAC cores 260 and one of the connectors 270, are used for each node to communicate with each other node with a dedicated connection 180.

These dedicated compute resources per dedicated connection 180 allow messages to be broadcast at exactly the same time by a sending node among other mesh nodes directly connected to it (e.g., via dedicated connections 180), particularly when GigE core 260 is implemented in hardware such as fixed logic device 230. Conversely, a broadcast message from a sending node can be received at exactly the same time by all receiving nodes. Each receiving node has its own dedicated connection 180 to the sending node, and reception is performed by dedicated connection communication logic and a dedicated interface for each node directly connected to the receiving node. Since each receiving node receives the broadcast message at exactly the same time, and internal message latency between different nodes in the mesh should be identical, no receiving node is preferred. Furthermore, a receiving node may receive different messages from different nodes directly connected to it at exactly the same time.

The ability to send and receive multiple messages completely simultaneously contrasts with other environments where communication between servers occurs through a switch, which typically requires that messages be serialized. When communication occurs through a switch, each server typically has a single connection (or possibly multiple redundant connections, which are still considered a single logical connection) between itself and the switch. If a server needs to broadcast a message to multiple other servers in the system, those messages are not actually sent to or received by the switch simultaneously, but instead must be sent/received serially, one after the other, over a single logical connection between the server and the switch. Similarly, even if a switch is capable of simultaneously receiving multiple different messages that are destined for the same server but originate from different originating servers (because the switch may have a single connection to each of the originating servers), these multiple different messages still must be serialized by the switch in order to send them one at a time over a single logical connection between the switch and the destination server. The message serialization required in such environments not only increases the overall latency of the system, but also leads to unpredictable latency between servers in such a system. For example, if a message needs to be broadcast from a server to 15 other servers in the system, those 15 messages will need to be serialized as they are sent to the switch, and the difference in latency can be significant depending on whether the message addressed to a particular destination server is the first or 15th message in a series of broadcast messages.

In addition to the latency benefits that can be gained from using dedicated connections 180 that do not require switch traversal, the point-to-point mesh architecture 172 enables further latency improvements. As discussed above in connection with FIG. 1B-1, messages being sent between two nodes in a point-to-point mesh, such as from gateway 120-1 to core compute node 140-1, can be sent through ordering path 117 to ensure that the messages can be processed throughout electronic trading system 100 in a deterministic order relative to other messages. As messages traverse ordering path 117, they pass through sequencer 150-1, which marks the messages with a unique order identifier and then generates an order-marked message that sequencer 150-1 sends to the destination node (in this example, core compute node 140-1). Upon receipt of the sequence-marked message, the destination node (in this example, core compute node 140-1) can ensure that the message is processed in the correct, deterministic order, as specified by the sequence identifier of the sequence-marked message relative to other messages in the electronic trading system 100 (which are also sequence-marked by the sequencer).

1B-1, this ordering path 117 may include multiple direct connections: a direct connection between a sending node and a sequencer (e.g., direct connection 180-gw1-s1 between gateway 120-1 and sequencer 150-1) and a further direct connection between a sequencer and a destination node (e.g., direct connection 180-c1-s1 between sequencer 150-1 and compute node 140-1). These multiple direct connections comprising ordering path 117, in some embodiments, do not traverse a switch and can benefit from the latency advantages already described above. Nevertheless, in such embodiments, the latency of ordering path 117 is affected by the message processing time at sequencer 150-1 (i.e., the time required for sequencer 150-1 to receive a non-order-marked (i.e., unordered) message, mark it with an order identifier, and send the order-marked message to its destination) and the transmission time of the message over the two direct connections. (i.e., the unordered version is first sent from the sender node to the sequencer via a direct connection, and then the ordered version is sent from the sequencer to the destination node via a separate direct connection.) Testing indicates that the latency impact of this ordering path 117 compared to a single direct connection is an additional 0.5-1.0 microseconds for the ordering path 117, depending on factors such as whether a custom protocol is used rather than a specific established protocol such as TCP/IP.

Thus, as an optimization to minimize the impact of potential latency in traversing messages through ordering path 117, unordered marked messages may also be transmitted in parallel over activation link 180-1-1, which may be a single direct connection between a sender node and a destination node, such as direct connection 180-1-1 between gateway 120-1 and core compute node 140-1. Thus, in some embodiments, unordered marked messages may be transmitted simultaneously by a sending node (e.g., gateway 120-1) to a destination node (e.g., core compute node 140-1) over activation link 180-1-1 and to sequencer 150-1 over a direct connection between the sending node and sequencer 150-1 (e.g., direct connection 180-gw1-s1) as part of ordering path 117. According to such an embodiment, sequencer 150-1 and a destination node (e.g., core compute node 140-1) can simultaneously receive the unordered version of the message, thereby enabling the destination node to immediately activate processing of the unordered message in response to its receipt. At the same time that the order-marked message is being generated by sequencer 150-1 and sent to the destination node, the destination node may begin processing the unordered message in parallel, which may include loading data associated with the stock symbol referenced in the message into fast-side memory, constructing a preliminary response message, or initiating matching function activities for electronic trading. As described above, upon receiving the order-marked message, the destination node may complete processing of the message in the proper deterministic order (i.e., order) as specified by the order identifier. However, by the time the destination node receives the order-marked message via ordering path 117, the destination node has already had an opportunity to perform useful processing on the unorder-marked version of the message (e.g., equivalent to 0.5 to 1.0 microseconds of processing time), allowing it to complete processing of the order-marked message and generate a response message with a corresponding lower latency than if the destination node had not yet received the unorder-marked version of the message via activation link 180-1-1.

Thus, as described above, point-to-point mesh architectures (e.g., 102 and 172) offer latency improvements in a number of ways. Through testing, at least some of these latency improvements have been empirically quantified. For example, latency improvements resulting from the use of direct connections between nodes, avoiding switches, can total at least 1.0 microseconds, and in some cases, up to 3.0 microseconds in each direction (i.e., inbound vs. outbound). Additionally, latency improvements resulting from the opportunity for the destination node to process unordered marked messages received over activation link 180-1-1 in parallel with the processing and transmission of messages over ordering path 117 can be an additional 0.5 to 1.0 microseconds in each direction. Thus, the overall potential latency improvement from the time an incoming message enters electronic trading system 100 from participant device 130 via gateway 120 to the time a corresponding response message is transmitted from gateway 120 to participant device 130 can be at least 2.5 to 7.0 microseconds. In latency-sensitive applications such as electronic trading, even a difference of a few microseconds in latency can be very significant. These latency improvements enable some embodiments of electronic trading system 100 to respond reliably according to desired response time latencies of 5.0 to 7.0 microseconds. This is a significant improvement over prior art trading systems, the best of which currently have response time latencies in the 50 to 75 microsecond range.

3 is a block diagram of another exemplary embodiment of a point-to-point mesh system 302. The point-to-point mesh system 302 includes a plurality of gateways 320, a plurality of core compute nodes 340, and a sequencer 350. Each gateway 320-1, 320-2, ..., 320-g of the plurality of gateways 320 is coupled to each core compute node 340-1, 340-2, ..., 340-c of the plurality of core compute nodes 340 via a respective first direct connection, i.e., first direct connection 380-a. The sequencer 350 is coupled to each gateway of the plurality of gateways 320 via a respective second direct connection, i.e., second direct connection 380-b, and to each core compute node of the plurality of core compute nodes via a respective third direct connection, i.e., third direct connection 380-c. 1B-1, 1B-2, 1D, and 3, gateway 120-1 may be a given gateway of multiple gateways 320 that are communicatively coupled to one another via a shared gateway network as disclosed above with respect to FIG. 1D or via direct connections 384a, 384b, ..., 384g of FIG. 3. Multiple gateways 320, in the exemplary embodiment, include gateways 320-1, 320-2, ..., 320-g. Core compute node 140-1 may be a given core compute node of multiple core compute nodes 340 that include core compute nodes 340-1, 340-2, ..., 340-c that may be communicatively coupled to one another via a shared core compute node network as disclosed above with respect to FIG. 1D or via direct connections 386a, 386b, ..., 386c of FIG. 3. It should be understood that the gateway number g may be the same as or different from the other number c of core compute nodes in point-to-point mesh system 302. According to an example embodiment, sequencer 150-1 may be a given sequencer among multiple sequencers communicatively coupled to one another via a shared sequencer network, such as sequencer-wide shared network 182-s disclosed above with respect to FIG. 1D, or via a direct connection, such as fourth direct connection 482 disclosed below with respect to FIG. 4.

Figure 4 is a block diagram of another example embodiment of a point-to-point mesh system 402. With reference to Figures 1B-2 and 4, sequencer 150-1 may be a given sequencer among multiple sequencers in point-to-point mesh system 402, including sequencer 450-1 and at least one other sequencer 450-2.

In the point-to-point mesh system 402, each gateway 420-1, 420-2, ..., 420-g of the plurality of gateways 420 is coupled to each core compute node 440-1, 440-2, ..., 440-c of the plurality of core compute nodes via a respective first direct connection, i.e., first direct connection 480a. Each gateway 420 is coupled to each sequencer 450-1, ..., 450-2 of the plurality of sequencers via a respective second direct connection, i.e., second direct connections 480-b-1 and 480-b-2. Each core compute node 440-1, 440-2, ..., 440-c of the plurality of core compute nodes 440 is coupled to each sequencer 450-1, ..., 450-2 of the plurality of sequencers via a respective third direct connection, i.e., third direct connection 480-c-1 and 480-c-2.

A given sequencer, i.e., a particular one of sequencers 450-1...450-2, may be the currently active sequencer serving the point-to-point mesh system, and each of the other sequencers of the plurality of sequencers may be a standby sequencer waiting to take over as the currently active sequencer. Each sequencer of the plurality of sequencers may be coupled to each of the other sequencers of the plurality of sequencers via a respective fourth direct connection, such as fourth direct connection 482. Each gateway 420-1, 420-2...420-g of the plurality of gateways 420 may be further configured to transmit a respective compute node-destined message (not shown) sent therefrom to a given one of the plurality of sequencers 450-1...450-2. Each core compute node 440-1, 440-2, ..., 440-c of the plurality of core compute nodes 440 is further configured to transmit a respective gateway-destined message (not shown) sent therefrom to a given sequencer among the plurality of sequencers 450-1, ..., 450-2. At least one other sequencer is in a standby state.

A standby state (standby role) may represent a passive state in which at least one other sequencer is configured to be powered on and in a "listen-only" mode. In the "listen-only" mode, at least one sequencer is configured to receive messages but not send messages and is capable of taking over as the active sequencer if the currently active sequencer fails. The currently active sequencer is in an active state (active role) capable of sending and receiving messages. Typically, a standby sequencer is a redundant (backup) sequencer that is powered on, capable of receiving messages, and capable of taking over as the active sequencer. Such a switch, i.e., from a standby state (role) to an active state (role), may be based on receipt of a switch command issued by a controller (not shown) due to a failure of the currently active sequencer or for other reasons, such as, but not limited to, a software update being applied to the currently active sequencer.

A given sequencer (in an active state) may further be configured to send order-marked messages, such as order-marked message 106' and order-marked response 107', to each of the other sequencers of the plurality of sequencers 450-1, ..., 450-2 via each respective fourth direct connection 482, so that a standby sequencer can take over as the currently active sequencer if the currently active sequencer fails or is no longer designated as the active sequencer for other reasons, such as, by way of non-limiting example, a software update. However, it should be understood that the active sequencer, i.e., the given sequencer in the active state, need not forward the order-marked messages to the standby sequencer, i.e., the sequencer in the standby state, but may instead continuously broadcast/replicate a journal, also referred to as a state log (which would include the order information therein), to the standby sequencer as disclosed above with respect to FIG. 1D.

The electronic trading system 100 may further include a system status log (not shown) as described above with reference to FIG. 1D. The active sequencer may be configured to transmit the system status log from the active sequencer to at least one other sequencer of the plurality of sequencers via a shared sequencer network. For example, the shared sequencer network may include a fourth direct connection 482, such that the system status log may be transmitted from sequencer 450-1 to sequencer 450-2 when sequencer 450-1 is the given sequencer in the active state.

FIG. 5 is a flow diagram 500 of an example embodiment of a method for executing electronic trading. The method begins (502) by sending a message representing an electronic trade request with a limit price to buy or sell a financial instrument from a gateway to a core compute node via a first direct connection (504). The message is then received by the core compute node for executing an electronic trading function in the electronic trading system. The method then sends the message from the gateway to a sequencer of the electronic trading system via a second direct connection (506), and in response, sends an order-marked message from the sequencer to the core compute node via a third direct connection (508). The first, second, and third direct connections have respective unshared bandwidths. The sequencer is inserted between the gateway and the core compute node via the second and third direct connections. The order-marked message sent by the sequencer is an ordered version of the message sent from the gateway via the second direct connection. The order-marked message is then received by the core compute node. The method includes receiving, at the core compute node, another message from the gateway, receiving an order-marked version of the other message from the sequencer, and determining, at the core compute node, a relative ranking of the order-marked message among the order-marked versions of the other messages received by the core compute node in the electronic trading system (510). The core compute node completes an electronic trade matching function for the electronic trade request in response to the determined relative ranking. This completion matches a limit price with a contra-limit price for the financial instrument, enabling electronic trading of the financial instrument. The method then ends (512) in the exemplary embodiment.

The message may be a gateway message. The method may further include receiving an incoming message from the participant device at the gateway. Sending the gateway message may include sending the gateway message by the gateway in response to receiving the incoming message by the gateway from the participant device. The order-marked message may be a first order-marked message. The method may further include sending the first order-marked message from the sequencer to the gateway via the second direct connection. In response to receiving the message, the first order-marked message may be received by the gateway. The method may further include sending a core compute node message from the core compute node to the gateway via the first direct connection, and sending the core compute node message to the sequencer via the third direct connection and in response sending a second order-marked message from the sequencer to the gateway via the second direct connection. The second order-marked message is an ordered version of the core compute node message. The method may further include determining at the gateway a relative ranking of the second order-marked message and order-marked versions of other messages sent from the core compute node to the gateway; sending an outgoing message to the participant device, the outgoing message being sent according to the determined relative ranking; and sending the second order-marked message from the sequencer to the core compute node via a third direct connection.

The gateway may be a given gateway among the plurality of gateways, and the core compute node may be a given core compute node among the plurality of core compute nodes. The method may further include transmitting a message destined for a respective compute node from each gateway of the plurality of gateways to all core compute nodes of the plurality of core compute nodes and the sequencer. The method may further include transmitting a message destined for a respective gateway from each core compute node of the plurality of core compute nodes to all gateways of the plurality of gateways and the sequencer, and transmitting each order-marked message from the sequencer to the plurality of gateways and the plurality of core compute nodes in response to the message destined for a respective compute node and the message destined for a respective gateway received at the sequencer.

The sequencer may be a given sequencer of a plurality of sequencers. The method may further include the steps of: sending a message destined for a respective compute node from each gateway of the plurality of gateways to the given sequencer; sending a message destined for a respective gateway from each core compute node of the plurality of core compute nodes to the given sequencer; and sending an order-marked message from the given sequencer to each other sequencer of the plurality of sequencers.

The method may further include receiving an identical message at multiple core compute nodes, the identical message being a message sent by a given gateway and destined for each compute node; generating response messages at the multiple core compute nodes in response to receiving the identical message; and receiving the response messages from among the multiple core compute nodes at the given gateway. The method may further include performing an action at the given gateway based on a given response message among the response messages generated in response to receiving the identical message. The given response message may be the first to arrive at the given gateway relative to other response messages among the response messages generated in response to receiving the same message. The method may further include ignoring other response messages that arrive at the given gateway after the given response message. Thus, the "same message" refers to a single message that is broadcast from a single gateway to all compute nodes, such that each compute node receives one copy of the single message. Multiple compute nodes may then each respond to the same single message, resulting in multiple functionally equivalent responses being broadcast to all gateways, such as being received by a given gateway. Gateways, including the given gateway, may then sort these multiple functionally equivalent messages in "first come, first served" order.

The method may include receiving, at a given compute node, a plurality of messages destined for the compute node from at least two of the plurality of gateways. Each of the plurality of messages destined for the compute node may represent a respective version of the same message, with each version having been input to a respective one of the at least two gateways via a respective high availability flow. The method may further include performing an action at the given compute node based on a message destined for the given compute node among the plurality of messages destined for the compute node. The message destined for the given compute node may be the first to arrive at the given compute node relative to messages destined for other compute nodes among the plurality of messages destined for the compute node. The method may further include ignoring messages destined for other compute nodes that arrive at the given compute node after the message destined for the given compute node.

The method may further include matching, at the core compute node, trade orders for the financial instruments based on the executed electronic trade matching function. The method may further include maintaining an open position for the financial instruments in the order book, the open position being an amount discrepancy for the financial instruments resulting from the executed electronic trade matching function.

The method may further include synchronizing the gateway, the core compute node, and the sequencer based on a clock.

The message may be a gateway message. The method may further include servicing at least one participant device at the gateway, receiving an incoming message at the gateway, and transmitting a gateway message from the gateway to the sequencer and the core compute node in response to receiving the incoming message at the gateway. The incoming message may have been dispatched by at least one participant device. The method may further include generating an order-marked message by marking the message or a representation thereof with a unique order identifier.

The method may further comprise protecting the first direct connection, the second direct connection, and the third direct connection, or some set thereof, via at least one respective redundant direct connection.

The gateway may be a given gateway among the plurality of gateways, the core compute node may be a given core compute node among the plurality of core compute nodes, and the sequencer may be a given sequencer among the plurality of sequencers. The method may further include enabling the plurality of gateways to communicate via a shared gateway network, enabling the plurality of core compute nodes to communicate via a shared core compute node network, and enabling the plurality of sequencers to communicate via the shared sequencer network or via their respective fourth direct connections.

The method may further include transmitting the system status log from the given sequencer to at least one other sequencer of the plurality of sequencers via a shared sequencer network, or storing the system status log by the given sequencer in a data store. The data store may be accessible to the plurality of sequencers via the shared sequencer network.

The electronic trading system may be an active electronic trading system. The method may include enabling at least one sequencer of the plurality of sequencers to communicate with a disaster recovery site. The disaster recovery site may include a standby electronic trading system. The standby electronic trading system may be a replica of the active electronic trading system and configured to allow electronic trading to continue if the active electronic trading system fails.

6 is a block diagram of an example embodiment of a sequencer 650 of an electronic trading system, such as the electronic trading system 100 disclosed above. In the example embodiment, the sequencer 650 comprises a first communication module 652 configured to communicate directly with a gateway 620 via a first direct connection 680a in a point-to-point mesh system, such as the point-to-point mesh system 102 of the electronic trading system 100 disclosed above. The sequencer 650 further comprises a second communication module 654 configured to communicate directly with a core compute node 640 via a second direct connection 680b in the point-to-point mesh system of the electronic trading system. The sequencer 650 further comprises sequencing logic 656 coupled to the first communication module 652 and the second communication module 654.

The ordering logic 656 is configured to generate an order-marked message 606' by marking the message 606, or a representation thereof, with a unique order identifier (not shown), and the message 606 is received by the first communication module 652 or the second communication module 654 via the first direct connection 680a or the second direct connection 680b, respectively. The ordering logic 656 is further configured to transmit the order-marked message 606' to the gateway 620 and the core compute node 640 via the first communication module 652 and the second communication module 654, respectively.

According to an exemplary embodiment, the sequencing logic is an ASIC or part of the logic of, for example, an FPGA in fixed logic device 230 of FIG. 2 disclosed above. Additionally, both communications modules 652 and 654 may be implemented in an FPGA, for example, in 10 GigE MAC core 260 disclosed above with reference to FIG. 2.

FIG. 7 is a block diagram of another example embodiment of an electronic trading system, such as the electronic trading system 100 of FIGS. 1B-1 and 1B-2 disclosed above. With reference to FIGS. 1B-2 and 7, the gateway 120-1, core compute node 140-1, sequencer 150-1, first direct connection 180-1-1, second direct connection 180-gw1-s1, and third direct connection 180-c1-s1 constitute a first point-to-point mesh system 702a. The electronic trading system 100 may be a first electronic trading system 700a communicatively coupled to a proxy node 772. The proxy node 772 may further be communicatively coupled to at least one participant device 730 and a second electronic trading system 700b. The second electronic trading system 700b may include a second point-to-point mesh system 702b, such as the point-to-point mesh system 102 disclosed above with respect to FIG. 1B-2.

The proxy node 772 is configured to transmit a message 703' to the first electronic trading system 700a and the second electronic trading system 700b in response to receiving an incoming message 703 from at least one participant device 730. The first and second electronic trading systems may be configured to generate respective responses to the message transmitted by the proxy node 772. The proxy node 772 may further be configured to transmit a response, i.e., an outgoing message 705, to at least one participant device 730 in response to receiving a first incoming response 705' received from the first electronic trading system 700a or the second electronic trading system 700b. The first electronic trading system 700a and the second electronic trading system 700b may be synchronized (777), for example, based on a common clock providing, for example, the date and time. However, the first electronic trading system 700a and the second electronic trading system 700b are not limited to being synchronized (777) based on a common clock.

According to an example embodiment, a single active/primary sequencer (not shown) for the overall system encompassing both electronic trading system 700a and electronic trading system 700b may be employed to coordinate sequence numbers between electronic trading system 700a and electronic trading system 700b. For example, the single active/primary sequencer may coordinate the sequence numbers so that arriving responses 705' are assigned the same sequence number by both electronic trading system 700a and electronic trading system 700b, thereby allowing proxy node 772 to properly determine the relative ordering of messages. For example, if there is a single active/primary sequencer in electronic trading system 700a, the active sequencer may be configured to communicate with electronic trading system 700b by communicating with a standby sequencer in electronic trading system 700b, and sequence numbers are assigned on a message-by-message basis.

The above-described architectures, such as a point-to-point mesh architecture, can be used for applications other than electronic trading systems. For example, they can be used to monitor data streams flowing through a network, capture packets, decode the raw data in the packets, analyze the packet contents in real time, and provide responses for applications other than handling securities trading orders.

Additionally, the exemplary embodiments disclosed herein may be configured using a computer program product, e.g., control may be programmed in software to implement the exemplary embodiments. Further exemplary embodiments may include a non-transitory computer-readable medium containing instructions executable by a processor, which, when loaded and executed, cause the processor to complete the methods described herein. It should be understood that elements of the block diagrams and flow diagrams may be implemented in software or hardware, such as by one or more arrangements of the circuitry of FIG. 2 disclosed above, or equivalents thereof, firmware, custom-designed semiconductor logic, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), combinations thereof, or other similar implementations determined in the future.

Furthermore, the elements of the block diagrams and flow diagrams described herein may be combined or divided in any manner in software, hardware, or firmware. If implemented in software, the software may be written in any language that can correspond to the example embodiments disclosed herein. The software may be stored in any form of computer-readable medium, such as one or more random access memories (RAMs), read-only memories (ROMs), or compact disc read-only memories (CD-ROMs). In operation, a general-purpose or application-specific processor or processing core loads and executes the software in a manner well understood in the art. Furthermore, it should be understood that the block diagrams and flow diagrams may include more or fewer elements, may be arranged or oriented in a different manner, or may be represented differently. It should be understood that examples may determine the number of block diagrams, flow diagrams, and/or network diagrams illustrating the implementation of the embodiments disclosed herein.

Accordingly, further embodiments may be implemented in a variety of computer architectures, physical, virtual, cloud computers, and/or any combination thereof, and therefore, the data processing systems described herein are for illustrative purposes only and are not intended to limit the embodiments.

While example embodiments have been particularly shown and described, it should be understood by those skilled in the art that various changes in form and detail can be made therein without departing from the scope of the embodiments encompassed by the appended claims.

Claims (30)

1. An electronic trading system comprising:
A gateway;
a core compute node configured to perform electronic trade matching functions;
A sequencer,
Equipped with
the gateway and the core compute node are coupled via a first direct connection, the gateway and the sequencer are coupled via a second direct connection, and the sequencer and the core compute node are coupled via a third direct connection, the first, second, and third direct connections having respective unshared bandwidths;
the gateway is configured to send a message representing an electronic transaction request with a limit price to buy or sell a financial instrument to the core compute node via the first direct connection, wherein the message is received by the core compute node; the gateway is further configured to send the message to the sequencer via the second direct connection, wherein the sequencer is configured to send an order-marked message to the core compute node via the third direct connection; the sequencer is interposed between the gateway and the core compute node via the second and third direct connections, wherein the order-marked message sent by the sequencer is an ordered version of the message sent by the gateway, whereby the order-marked message is received by the core compute node, the core compute node being configured to determine a relative ranking of the order-marked message among order-marked versions of other messages received by the core compute node in the electronic trading system, and the core compute node being further configured to complete the electronic trade matching function for the electronic trade request in response to the determined relative ranking, to match the limit price with a counterparty limit price for the financial instrument, and to enable electronic trading of the financial instrument. The electronic trading system of claim 1, wherein the message and the sequence-marked message contain the same user data, and the user data is associated with the electronic trading request. the message is a gateway message sent by the gateway in response to receiving an incoming message received by the gateway from a participant device, the sequencer is further configured to, in response, send the order-marked message to the gateway via the second direct connection, whereby the order-marked message was received by the gateway, the order-marked message is a first order-marked message, and the core compute node is further configured to:
configured to, in response to receiving the gateway message, send a core compute node message to the gateway via the first direct connection;
the core compute node is configured to send the message to the sequencer via the third direct connection, the sequencer being further configured to in response send a second order-marked message to the gateway via the second direct connection, the second order-marked message being an ordered version of the core compute node message, and the gateway being further configured to:
configured to determine the relative ranking of the second order-marked message and order-marked versions of other messages sent from the core compute node to the gateway;
2. The electronic trading system of claim 1, wherein the sequencer is configured to send an outgoing message to the participant device, the outgoing message being sent according to the determined relative ranking, and the sequencer is further configured to in response send the second order-marked message to the core compute node via the third direct connection. the gateway is a given gateway among a plurality of gateways, and the core compute node is a given core compute node among a plurality of core compute nodes;
each of the plurality of gateways is coupled to a respective one of the plurality of core compute nodes via a respective one of the first direct connections;
the sequencer is coupled to each of the plurality of gateways via the respective second direct connections and to each of the plurality of core compute nodes via the respective third direct connections, the plurality of gateways, the plurality of core compute nodes, the sequencer, and the respective direct connections forming at least a portion of a point-to-point mesh system, within the point-to-point mesh system:
Each of the plurality of gateways is configured to transmit a message destined for a respective compute node sent therefrom to all of the plurality of core compute nodes and to the sequencer; and each of the plurality of core compute nodes is configured to transmit a message destined for a respective gateway sent therefrom to all of the plurality of gateways and to the sequencer.
2. The electronic trading system of claim 1, wherein the sequencer is further configured to transmit each of the sequence-marked messages to the plurality of gateways and the plurality of core compute nodes in response to receiving a message destined for the respective compute node or a message destined for the respective gateway. the sequencer is a given sequencer among a plurality of sequencers in the point-to-point mesh system;
each of the plurality of gateways is coupled to a respective one of the plurality of sequencers via a respective one of the second direct connections;
each core compute node of the plurality of core compute nodes is coupled to a respective sequencer of the plurality of sequencers via a respective third direct connection;
the given sequencer is a currently active sequencer serving the point-to-point mesh system, and each other sequencer of the plurality of sequencers is a standby sequencer waiting to take over for the currently active sequencer;
each said sequencer of said plurality of sequencers is coupled to each other sequencer of said plurality of sequencers via a respective fourth direct connection;
each of the plurality of gateways further configured to send a message destined for a respective compute node to the given sequencer of the plurality of sequencers;
each core compute node of the plurality of core compute nodes further configured to send a respective gateway-destined message to the given sequencer of the plurality of sequencers;
5. The electronic trading system of claim 4, wherein the given sequencer is further configured to send the order-marked message to each other sequencer of the plurality of sequencers via each respective fourth direct connection to enable the standby sequencer to take over for the currently active sequencer if the currently active sequencer fails. The message destined for each of the compute nodes transmitted by the given gateway is the same message received by the plurality of core compute nodes, the plurality of core compute nodes are configured to generate a response message in response to receiving the same message, the response message being received at the given gateway from among the plurality of core compute nodes, and the given gateway further:
and configured to take action based on a given one of the response messages generated in response to receipt of the same message, the given response message being a first to arrive at the given gateway relative to other response messages generated in response to receipt of the same message, the given gateway further comprising:
5. The electronic trading system of claim 4, configured to ignore the other response messages that arrive after the given response message. Messages destined for multiple compute nodes representing the same message are received from among the multiple gateways at a given compute node, and the given compute node further comprises:
configured to take action based on a message destined for a given compute node among the plurality of messages destined for the given compute node, the message destined for the given compute node arriving at the given compute node first relative to other messages destined for the plurality of messages representing the same message, the given compute node further comprising:
5. The electronic trading system of claim 4, configured to ignore messages destined for the other compute nodes that arrive after a message destined for the given compute node. an order book accessible by the core compute nodes, the core compute nodes further comprising:
configured to match trade orders for the financial instruments using the electronic trade matching functionality;
2. The electronic trading system of claim 1, configured to maintain a balance of the financial instrument in the order book, wherein a mismatch amount of a trading order associated with the financial instrument results from performing the electronic trade matching function, and wherein the balance includes a mismatch amount of a trading order associated with the financial instrument. The electronic trading system of claim 1, further comprising a clock, wherein the gateway, the core compute node, and the sequencer are synchronized based on the clock. The gateway further comprises:
configured to provide a service to at least one participant device;
2. The electronic trading system of claim 1, wherein the gateway is configured to send an incoming message to the sequencer and the core compute node in response to receiving the message, the incoming message being dispatched by the at least one participant device, and the sequencer is further configured to generate the sequence-marked message by marking the message with a unique sequence identifier or by creating a representation of the received message, marking the representation with the unique sequence identifier, and transmitting the marked representation, wherein the marked representation is the sequence-marked message. The electronic trading system of claim 1 further comprising at least one redundant direct connection for each of the first direct connection, the second direct connection, and the third direct connection, or a subset thereof. the gateway is a given gateway among a plurality of gateways communicatively coupled to each other via a shared gateway network;
the core compute node is a given core compute node among a plurality of core compute nodes communicatively coupled to each other via a shared core compute node network;
2. The electronic trading system of claim 1, wherein the sequencer is a given sequencer of a plurality of sequencers communicatively coupled to one another via a shared sequencer network or via respective fourth direct connections. The electronic trading system of claim 12, further comprising a system status log, wherein the given sequencer is configured to transmit the system status log to at least one other sequencer among the plurality of sequencers via the shared sequencer network or to store the system status log in a data store, the data store being accessible to the plurality of sequencers via the shared sequencer network. The electronic trading system of claim 12, wherein the electronic trading system is an active electronic trading system, and at least one of the plurality of sequencers is communicatively coupled to a disaster recovery site, the disaster recovery site including a standby electronic trading system, the standby electronic trading system being a replica of the active electronic trading system and configured to enable electronic trading to continue in the event of a failure of the active electronic trading system. the gateway, the core compute node, the sequencer, and the first, second, and third direct connections form a first point-to-point mesh system;
the electronic trading system is a first electronic trading system communicatively coupled to a proxy node, the proxy node further communicatively coupled to at least one participant device and a second electronic trading system, the second electronic trading system including a second point-to-point mesh system;
2. The electronic trading system of claim 1, wherein the proxy node is configured to transmit an incoming message to the first and second electronic trading systems in response to receiving the message from the at least one participant device, the first and second electronic trading systems being configured to generate respective responses to the message transmitted by the proxy node, and the proxy node is further configured to transmit a response to the at least one participant device in response to receiving a first-arriving response from the respective responses generated and received from the first or second electronic trading systems. 1. A method of conducting electronic transactions, comprising:
sending a message from the gateway to a core compute node over a first direct connection representing an electronic transaction request with a limit price to buy or sell a financial instrument;
In response, receiving the message at the core compute node for performing an electronic trading function in an electronic trading system;
sending the message from the gateway to a sequencer in the electronic trading system via a second direct connection, and in response sending an order-marked message from the sequencer to the core compute node via a third direct connection, wherein the first, second, and third direct connections have respective unshared bandwidths, the sequencer is interposed between the gateway and the core compute node via the second and third direct connections, and the order-marked message sent by the sequencer is an ordered version of the message sent from the gateway via the second direct connection;
In response, receiving the marked message at the core compute node;
receiving, at the core compute node, another message from the gateway and receiving, at the core compute node, an order-marked version of the other message from the sequencer;
determining at the core compute node a relative ranking of the order-marked message among the order-marked versions of the other messages received by the core compute node in the electronic trading system;
at the core compute node, completing an electronic trade matching function for the electronic trade request in response to the determined relative ranking, the completing matching the limit price with a counterparty limit price for the financial instrument to enable electronic trading of the financial instrument;
A method for providing The method of claim 16, wherein the message and the sequence-marked message contain the same user data, and the user data is associated with an electronic transaction request. the message is a gateway message, the method further comprising receiving an incoming message at the gateway from a participant device, wherein sending the gateway message comprises sending the gateway message by the gateway in response to receiving the incoming message from the participant device by the gateway, the order-marked message is a first order-marked message, and the method further comprises:
sending the first order-marked message from the sequencer to the gateway via the second direct connection, whereupon the first order-marked message is received by the gateway;
In response to receiving the message, sending a core compute node message from the core compute node to the gateway via the first direct connection;
sending the core compute node message to the sequencer over the third direct connection and in response sending a second order-marked message from the sequencer to the gateway over the second direct connection, the second order-marked message being an ordered version of the core compute node message;
determining at the gateway the relative ranking of the second order-marked message and the order-marked versions of other messages sent from the core compute node to the gateway;
sending an outgoing message to the participant device, the outgoing message being sent according to the determined relative ranking;
sending the second order-marked message from the sequencer to the core compute node via the third direct connection;
17. The method of claim 16 further comprising: The gateway is a given gateway among a plurality of gateways, and the core compute node is a given core compute node among a plurality of core compute nodes, and the method includes:
sending a message addressed to a respective compute node from each of the plurality of gateways to all of the core compute nodes of the plurality of core compute nodes and the sequencer;
sending a respective gateway-destined message sent from each of the core compute nodes of the plurality of core compute nodes to all of the gateways of the plurality of gateways and to the sequencer;
transmitting each of the order-marked messages from the sequencer to the plurality of gateways and the plurality of core compute nodes in response to the respective compute node-destined messages and the respective gateway-destined messages received at the sequencer;
17. The method of claim 16 further comprising: the sequencer is a given sequencer of a plurality of sequencers, and the method comprises:
sending a message destined for a respective compute node from each of the plurality of gateways to the given sequencer;
sending the respective gateway-destined messages sent from each core compute node of the plurality of core compute nodes to the given sequencer;
transmitting the sequence-marked message from the given sequencer to each other of the plurality of sequencers;
20. The method of claim 19 further comprising: receiving a same message at the plurality of core compute nodes, the same message being a message sent by the given gateway and destined for the respective compute node;
generating a response message at the plurality of core compute nodes in response to receiving the same message;
receiving the response message from among the plurality of core compute nodes at the given gateway;
performing an action at the given gateway based on a given one of the response messages generated in response to receiving the same message, the given response message being the first to arrive at the given gateway relative to other one of the response messages generated in response to receiving the same message;
ignoring the other response messages that arrive at the given gateway after the given response message;
20. The method of claim 19 further comprising: receiving, at a given compute node, a plurality of messages destined for the compute node from among the plurality of gateways, the plurality of messages destined for the compute node representing the same message;
performing an action at the given compute node based on a message destined for the given compute node among the plurality of messages destined for the given compute node, the message destined for the given compute node arriving at the given compute node first relative to other messages destined for the given compute node among the plurality of messages destined for the same message;
ignoring messages destined for the other compute nodes that arrive at the given compute node after the message destined for the given compute node;
20. The method of claim 19 further comprising: matching, at the core compute nodes, trade orders for the financial instruments based on the electronic trade matching function being performed;
maintaining a balance of said financial instrument in an order book, said balance being the discrepancy amount of trade orders related to said financial instrument resulting from said electronic trade matching function being executed;
17. The method of claim 16 further comprising: The method of claim 16, further comprising synchronizing the gateway, the core compute node, and the sequencer based on a clock. the message is a gateway message, and the method comprises:
providing a service to at least one participant device at the gateway;
receiving an incoming message at the gateway;
sending the gateway message from the gateway to the sequencer and the core compute node in response to receiving the incoming message at the gateway, the incoming message having been sent by the at least one participant device;
generating the sequence-marked message by marking the message or a representation thereof with a unique sequence identifier;
17. The method of claim 16 further comprising: The method of claim 16, further comprising protecting the first direct connection, the second direct connection, and the third direct connection, or some set thereof, via at least one respective redundant direct connection. the gateway is a given gateway of a plurality of gateways, the core compute node is a given core compute node of a plurality of core compute nodes, and the sequencer is a given sequencer of a plurality of sequencers, and the method comprises:
enabling the plurality of gateways to communicate via a shared gateway network;
enabling the plurality of core compute nodes to communicate over a shared core compute node network;
enabling the plurality of sequencers to communicate via a shared sequencer network or via respective fourth direct connections;
17. The method of claim 16 further comprising: The method of claim 27, further comprising a step of transmitting a system status log from the given sequencer to at least one other sequencer of the plurality of sequencers via the shared sequencer network, or storing the system status log by the given sequencer in a data store, the data store being accessible to the plurality of sequencers via the shared sequencer network. the electronic trading system is an active electronic trading system, and the method comprises:
28. The method of claim 27, further comprising enabling at least one of the plurality of sequencers to communicate with a disaster recovery site, the disaster recovery site including a standby electronic trading system, the standby electronic trading system being a replica of the active electronic trading system and configured to allow electronic trading to continue if the active electronic trading system fails. 1. A sequencer for an electronic trading system, comprising: a non-transitory computer readable medium encoding a set of instructions, the set of instructions, when loaded into the sequencer and executed, causing the sequencer to:
directly communicating with a gateway via a first direct connection in a point-to-point mesh system of the electronic trading system;
communicating directly with a core compute node via a second direct connection in the point-to-point mesh system of the electronic trading system, the sequencer being interposed between the gateway and the core compute node via the first and second direct connections, the first and second direct connections having respective unshared bandwidths;
A sequencer that generates an order-marked message by marking a message or a representation thereof with a unique order identifier, the message being received by the sequencer via the first or second direct connection, respectively, and the set of instructions further causes the sequencer to transmit the order-marked message to the gateway and the core compute node.