KR100834431B1 - RNC-based offload of iSCSI data transfer function by initiator - Google Patents

Abstract

The Remote-direct-memory-access-enabled Network Interface Controller (RNIC) mechanism used for the Remote Direct Memory Access (RDMA) function implements the Internet Small Computer System Interface (SCSI) offload initiator function. Methods and systems are provided that include steps.

Communication protocol, RDMA, RNIC, iSCSI,

Description

RNIC-based OFFLOAD OF iSCSI DATA MOVEMENT FUNCTION BY INITIATOR}

1 is a simplified flowchart of SCSI write and SCSI read transactions.

2 is a simplified flowchart of the iSCSI protocol showing sequencing rules and SCSI commands.

3 is a simplified block diagram illustrating a distributed computer system in accordance with an embodiment of the present invention.

4 is a simplified block diagram illustrating an RDMA mechanism implementing iSCSI offload functionality in accordance with an embodiment of the present invention.

5 is a simplified flowchart of remote memory access operations of RDMA, read and write.

6 is a simplified flowchart of memory registration in RDMA that may enable accessing system memory both locally and remotely, in accordance with an embodiment of the present invention.

7 and 8 are simplified block diagrams and flowcharts illustrating offloading of iSCSI data movement operations by RDMA supporting RNIC, respectively, in accordance with an embodiment of the present invention.

9 is a simplified block diagram illustrating a software architecture implemented using RDMA-based iSCSI offload, in accordance with an embodiment of the present invention.

10 is a simplified flowchart of direct data placement of an iSCSI data movement PDU into a SCSI buffer without hardware / software interaction, in accordance with an embodiment of the present invention.

11A and 11B illustrate processing of Data-In and Requested Data-Out by an RNIC and an iSCSI pay carried by the PDU according to an embodiment of the present invention. Simplified flowchart of performing load data placement directly into the registered SCSI buffer.

12 is a simplified flowchart of processing incoming R12T hardware and generating a Data-Out PDU, in accordance with an embodiment of the present invention.

<Description of the symbols for the main parts of the drawings>

201: iSCSI operation

202: SCSI command

301: host processor node

306: Computer program product

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention generally relates to a communication protocol between a host computer and an input / output (I / O) device, and more particularly, to an Internet Small Computer System Interface (RSI) by RDMA (Remote Direct Memory Access). ) Offload implementaion.

RDMA is a technology for efficient movement of data through high speed transmission. RDMA allows one computer to place information directly into another computer's memory with minimal memory bus bandwidth and CPU processing overhead while maintaining memory protection semantics. RNIC is a network interface card that provides RDMA services to consumers. The RNIC may provide support for RDMA via a transport control protocol (TCP).

One of the many important features of an RNIC is that it can function as an iSCSI target or initiator adapter. iSCSI defines the term initiator and target as follows. That is, an "initiator" refers to a SCSI command requester (eg, a host), and a "target" refers to an I / O such as a SCSI command responder (eg, a SCSI drive carrier, tape, etc.). Device). RNIC also provides iSER ("iSCSI Extentions for RDMA") services. iSER is an extension of iSCSI's data transfer model, allowing the iSCSI protocol to take advantage of the direct data placement technology of the RDMA protocol. The iSER data transfer protocol allows iSCSI implementations with RNIC to have data transfers that achieve true zero copy behavior by eliminating TCP / IP processing overhead while maintaining compatibility with the iSCSI infrastructure. iSER uses the RDMA wire protocol and is not transparent to the remote side (target or initiator). iSER also makes minor changes or modifications to the iSCSI implementation over RDMA. For example, iSER eliminates iSCSI Protocol Data Units (PDUs) such as DataOut and DataIn and uses RDMA Read and RDMA Write messages instead. Basically, iSER provides iSCSI-like functionality to the upper layers, but differs in the protocol and wire protocol of data movement.

The iSCSI protocol exchanges iSCSI PDUs to execute the SCSI commands provided by the SCSI layer. The iSCSI protocol allows for a smooth transition from locally attached SCSI storage to remotely attached SCSI storage. The iSCSI service may provide partial offload of the iSCSI function, and the level of offload may be implementation dependent. In summary, iSCSI uses regular TCP connections, while iSER implements iSCSI over RDMA. iSER uses other RDMA features to use RDMA connections, achieve better recovery, and improve latency and performance. Because RNIC supports both iSCSI and iSER services, RNIC enables SCSI communication with devices that support different levels of iSCSI implementation. Protocol selection (iSCSI vs. iSER) is made at the iSCSI login stage.

RDMA uses an operating system programming interface called "verb" to place a work request (WR) on a work queue. An example of implementing an iSER with a work request is described in US Patent Application No. 20040049600 to Boyd et al. Assigned to IBM. In this application, a work request comprising an iSCSI command may be received from a host to a network offload engine, and in response to receiving the work request, a memory area associated with the host may be registered in a translation table. . As in RDMA, work requests may be received via send queues, and in response to registering the memory area, a completion queue element may be placed on the completion queue.

It is an object of the present invention to provide an efficient iSCSI offload implementation by RNIC as described in more detail herein below and to achieve this offload level using the RNIC mechanism developed for RDMA.

According to the present invention, the iSCSI offload function may be implemented with an readily available RNIC mechanism used for the RDMA function. This involves remote direct data placement of the data-in and data-out payloads into the pre-registered SCSI buffers in any order, in accordance with any SCSI buffer offset, as in the case of an RDMA write operation. It includes, but is not limited to. Incoming R2T ("ready to transfer") PDUs may be processed, and data-out PDUs may be generated using the same mechanism as in the case of an RDMA read request. Controlling iSCSI PDUs can be deployed, for example, using receive queues and shared receive queues.

According to one aspect of the invention, an Internet Small Computer System Interface (SCSI) offload initiator function with a Remote-direct-memory-access-enabled Network Interface Controller (RNIC) mechanism used for Remote Direct Memory Access (RDMA) functionality. A method is disclosed that includes implementing an offload initiator function.

According to a second aspect of the present invention, an Internet Small Computer System Interface (SCSI) offload initiator with a Remote-direct-memory-access-enabled Network Interface Controller (RNIC) mechanism used for Remote Direct Memory Access (RDMA) functionality. A computer program product is disclosed that includes instructions for implementing an offload initiator function.

According to a third aspect of the present invention, an RDMA Service Unit, an inbound and an outgoing RDMA message is processed, and a direct deployment and delivery operation is performed using a service provided by the RDMA Service Unit. An RDMA messaging unit operative to perform, and an iSCSI messaging unit operative to perform an iSCSI offload initiator function and to handle incoming and outgoing iSCSI PDUs, wherein the iSCSI messaging unit comprises the RDMA service; A system is disclosed that is configured to perform direct placement and delivery of an iSCSI payload carried by the PDU to a registered SCSI buffer using a service provided by a unit.

The invention will be more fully understood from the following detailed description set forth in conjunction with the accompanying drawings.

In order to better understand the present invention, a general description of the iSCSI data movement and offload functionality is now provided (with reference to FIGS. 1 and 2). Subsequently, implementation of the iSCSI data movement and offload functionality in a distributed computer system with RDMA verbs and mechanisms is described (with reference to FIGS. 4 and subsequent figures).

The iSCSI protocol exchanges iSCSI protocol data units (PDUs) to execute the SCSI commands provided by the SCSI layer. The iSCSI protocol allows for a smooth transition from locally attached SCSI storage to remotely attached SCSI storage.

There are two main groups of iSCSI PDUs: iSCSI Control and iSCSI Data Movement PDUs. iSCSI control defines many types of control PDUs, especially SCSI commands, SCSI responses, and task management requests. Data transfer PDUs are smaller groups that include, but are not limited to, ready to transfer (R2T), SCSI data-out (requested and unsolicited), and SCSI data-in PDUs.

As mentioned above, "initiator" refers to a SCSI command requester (eg, a host) and "target" refers to a SCSI command responder (eg, an I / O device such as a SCSI driver carrier, tape, etc.). All iSCSI control and data movement commands can be divided into those generated by the initiator and processed by the target and those generated by the target and processed by the initiator.

Referring now to FIG. 1, the flow of SCSI write and SCSI read transactions is illustrated respectively.

In the SCSI write flow, the initiator sends a SCSI write command (denoted by reference numeral 101) to the target. This command passes an initiator task tag (ITT) that identifies the SCSI buffer that must be placed on disk (or other part of the target), among other fields. The SCSI write command can also carry immediate data, the maximum size of which can be negotiated at the iSCSI logic stage. In addition, the SCSI write command may be followed by a so-called unsolicited data-out PDU. Unsolicited data-out PDUs are identified by a target transfer tag (TTT), in which case the TTT must be 0xFFFFFFFF. The size of unsolicited data is also negotiated during the iSCSI login phase. These two data transfer modes can reduce latency for short SCSI write operations, but this can also be used to transfer the initial amount of data in large transactions. The maximum data size that can be transmitted in unsolicited or immediate mode depends on the buffering capability of the target.

After the target receives the SCSI write command, the target responds with one or more R2Ts (denoted by reference numeral 102). Each R2T indicates that the target is ready to receive the specified amount of data from the specified offset in the SCSI buffer (not necessarily in order). R2T carries two tags, an ITT from the SCSI command and a TTT pointing to the target buffer where the data is to be placed.

For each received R2T, the initiator may send one or more data-out PDUs (denoted by reference numeral 103). The data-out PDU carries data from the SCSI buffer [indicated by ITT]. Each received data-out carries a TTT indicating where to place the data. The last received data-out also carries an F-bit (denoted by reference numeral 104). This bit indicates that the last data-out has been received, which indicates to the target that the R2T exchange is complete.

When the target is notified that all R2T is complete, the target sends a SCSI response PDU (denoted by reference numeral 105). The SCSI response conveys the ITT and indicates that the SCSI write operation has completed successfully.

In the SCSI read flow, the initiator sends a SCSI read command to the target (denoted by reference numeral 106). This command passes an ITT that identifies the SCSI buffer from which to read data, among other fields.

The target may respond to one or more data-in PDUs (indicated by reference numeral 107). Each data-in carries data to be placed in a SCSI buffer. The data-ins may come in any order and may have any size. Each data-in carries an ITT that identifies the SCSI buffer and buffer offset where the data is to be placed.

A SCSI response follows the stream of data-in PDUs (indicated by reference numeral 108). The SCSI response conveys an ITT indicating whether the SCSI read operation completed successfully.

Note that, according to one embodiment of the present invention, unlike the prior art, the RNIC handles the flow of data-out and data-in and R2T.

Referring now to FIG. 2, an example of an iSCSI protocol is shown. The iSCSI protocol has well defined sequencing rules. The iSCSI task (denoted by reference numeral 210) includes one or more SCSI commands 202. At any given time, the iSCSI task 201 may have a single raw instruction 202. Each job 201 is identified by ITT 203. A single iSCSI connection can have multiple raw iSCSI tasks. The PDU 204 of the iSCSI task 201 may be interleaved in the connection stream. Each iSCSI PDU 204 may carry several sequence numbers. Sequence numbers associated with data movement PDUs include, but are not limited to, R2TSN (R2T Sequence Number), DataSN and ExpDataSN, and StatSN and ExpStatSN.

The DataSN is carried by each iSCSI PDU 204 carrying data (data-out and data-in). In the case of data-in, the DataSN may start at zero for each SCSI read command and may be incremented by each target with each transmitted data-in. The SCSI response PDU following the data-in carries an ExpDataSN indicating the number of data-ins transmitted for each individual SCSI command. For bidirectional SCSI commands, DataSN is shared by data-in and R2T, where R2T carries R2TSN instead of DataSN, but they share the same location within the BHS-Buffer Segment Handle Stack (buffer segment handle stack). Have different names for the same field.

For data-out, the DataSN can start at 0 for each R2T and be incremented by the initiator with each data-out sent. R2TSN may be carried by R2T. The R2TSN can start at zero for each SCSI write command and can be incremented by the target with each R2T sent.

Both DataSN and R2TSN can be used to follow the order of received data movement PDUs. Note that iSCSI allows out of order placement of received data and out of order execution of R2T. However, iSCSI requires an implementation to prevent the deployment of already deployed data or the execution of R2T already executed for the initiator and target.

StatSN and ExpStatSN can be used in the management of the target response buffer. The target may increment StatSN with each generated response. The response and potentially the data used in that command may be retained in the internal target until the initiator confirms receipt of the response using ExpStatSN. ExpStatSN may be delivered by all iSCSI PDUs flowing in the direction from the initiator to the target. The Initiator may monotonically increase ExpStatSN to allow efficient implementation of the target.

As mentioned above, in accordance with a non-limiting embodiment of the present invention, the iSCSI offload function may be implemented with the RNIC mechanism used for the RDMA function. First, a general description of the concept of work queues in RDMA for use in distributed computer systems is now described.

Referring now to FIG. 3, illustrated is a distributed computer system 300 in accordance with one embodiment of the present invention. Distributed computer system 300 may include, but is not limited to, an Internet Protocol network (IP net) and many other computer systems of various other types and configurations. For example, a computer system implementing the present invention can range from a small server with one processor and several input / output (I / O) devices to a massively parallel supercomputer system with multiple processors and I / O adapters. have. In addition, the present invention may be implemented in the infrastructure of remote computer systems connected by the Internet or an intranet.

The distributed computer system 300 may connect any number of host processor nodes 301 and any type, including but not limited to independent processor nodes, storage nodes, and special purpose processing nodes. Any of the nodes can function as an endnode, which is defined herein as a device that sends or finally consumes a message or frame in distributed computer system 300. Each host processor node 301 may include a consumer 302, which is a process running on that host processor node 301. The host processor node 301 may also include one or more IP Suite Offload Engines (IPSOEs) that may be implemented in hardware or in a combination of hardware and offload microprocessor (s). Offload engine 303 may support multiple queue pairs 304 used to deliver messages to IPSOE port 305. Each queue pair 304 may include a send work queue (SWQ) and a receive work queue (RWW). The transmit work queue can be used to send channel and memory semantic messages. The receive work queue may receive a channel semantic message. The consumer can use a "verb" that defines the semantics needed to implement to place a work request (WR) on the work queue. The verb also provides a mechanism to retrieve completed work from the completion queue.

For example, a consumer can create a work request that is placed as a work queue element (WQE) on a work queue. Thus, the transmit work queue may include a WQE that describes data to be transmitted over the fabric of distributed computer system 300. The receive work queue may include a WQE that describes where to place incoming channel semantic data from the fabric of distributed computer system 300. The work queue element may be processed by software or hardware within the offload engine 303.

The completion queue may include a completion queue element (CQE) that contains information about previously completed work queue elements. Completion queues can be used to generate points in time or points of completion notification for multiple queue pairs. A completion queue element is a data structure for a completion queue that contains enough information to determine a particular work queue element and queue pair that has been completed. A completion queue context is a block of information that includes a length and a pointer to other information needed to manage an individual completion queue.

The RDMA read operation request provides a memory semantic operation for reading a nearly continuous memory space on the remote node. The memory space may be part of the memory area or part of a memory window. A memory region refers to a series of previously registered nearly consecutive memory addresses defined by virtual addresses and lengths. A memory window is a series of nearly contiguous memory addresses that are bound to previously registered regions. Similarly, the RDMA write work queue element provides a memory semantic operation for writing to nearly contiguous memory space on the remote node.

Bind (unbind) The remote access key (Stag-Steering Tag) job queue element associates (or disassociates) a memory window with the memory area by the offload engine hardware. Provide commands to modify (or destroy). The STag is part of each RDMA access and is used to verify that the remote process has allowed access to the buffer.

It should be noted that the methods and systems shown and described below may include instructions for implementing the methods and systems described herein, network interface cards, hard disks, optical disks, memory devices, and the like, but not limited thereto. May be implemented by a computer program product 306).

Some related RDMA mechanisms for implementing the iSCSI offload function are now described with reference to FIG. 4.

In RDMA, Host A can access Host B's memory without any involvement of Host B. Host A decides where and when to access Host B's memory, and Host B does not know that this access occurs unless Host A provides explicit notification.

Before Host A can access Host B's memory, Host B must register the memory area to be accessed. Each registered memory area receives a STag. The STag is associated with an entry in a Protection Table called a Protection Block (PB). The PB details the registered memory area, including its boundaries, access rights, and so forth. RDMA allows for physically discontinuous memory areas. This area is represented by a page-list (or block-list). PB also points to a memory area page-list (or block-list).

RDMA can only remotely access registered memory areas. The memory area STag is used by the remote side to refer to that memory when accessing it. In storage applications, RDMA accesses memory areas with zero-based access. In zero-based access, the target offset (TO) delivered by the tagged tagged data placement protocol (DDP) segment defines the offset in the registered memory area.

Referring now to FIG. 5, the remote memory access operation of RDMA, i.e., read and write, is described. The remote write operation can be implemented using an RDMA write message-a tagged DDP message-that carries data that must be placed in the remote memory (denoted by reference numeral 501).

The remote read operation can be implemented using two RDMA messages-an RDMA read request message and an RDMA read response message-indicated by reference numeral 502. RDMA reads are Untagged DDP messages that specify both where data should be fetched and where to place data. The RDMA read response is a tagged DDP message that carries the data requested by the RDMA read request.

The process of processing the incoming tagged DDP segment (which is used for both RDMA write and RDMA read response) involves reading the PB referred to by the STag (503), access verification (504), area page-list (conversion table). A read operation 505 and a direct write operation 506 to memory. Incoming RDMA read requests may be queued by the RNIC (507). This queue is called the ReadResponse WorkQueue.

The RNIC may process the RDMA read request in order after all preceding RDMA requests have been completed (508) and may generate (509) an RDMA read response message sent back to the requestor.

The process of processing an RDMA read request may include queuing and dequeuing an optional RDMA read request with a read response work queue (510), the data source STag (which is the area of memory where the data should be read). Read PB (511), access verification 512, read region page-list (conversion table) (513), and read directly from memory and generate RDMA read response segments Step 514 may be included, but is not limited thereto.

RDMA defines an Address Translation and Protection (ATP) mechanism that enables local and remote access to system memory. This mechanism is based on the registration of the memory to be accessed, as will now be described with reference to FIG. 6.

Memory registration is an essential operation required for remote memory access. Two methods can be used in RDMA: Memory Window and Fast Memory Registration.

The memory window method (reference number 600) can be used when the memory to be accessed remotely is static and knows in advance which memory is to be accessed (601). In that case, the memory area is registered using a so-called classical memory registration scheme, and the allocation and updating of PB and TT (Translation Table) is performed by the driver with or without hardware assistance (602). This is a synchronous operation that can only be completed when both PB and TT are updated with their respective information. The memory window is used to allow (or inhibit) remote memory access to all (or a portion) of the registered memory area (603). This process is called window binding and is performed by the RNIC at the consumer request. This is much faster than memory registration. However, memory windows are not the only way to allow remote access. The STag of the area itself can also be used for this purpose. Thus, three mechanisms can be used to access registered memory: using statically registered regions, using windows bound to these regions, and / or using fast registered regions. .

If the memory for remote access is not known beforehand (604), use of a pre-registered area is not efficient. Instead, RDMA defines a fast memory registration and invalidation method 605.

This method keeps the memory registration process in two parts: the allocation 606 of RNIC resources to be consumed by the region (e.g., the portion of the PB and TT used to hold the page-list), and region-specific information. To update 607 of PB and TT. The first operation 606 may be performed by software and may be performed once for each STag. The second operation 607 may be posted by software and performed by hardware, and may be performed multiple times (for each new area / buffer to be registered). In addition to fast memory registration, RDMA defines an Invalidate operation that allows to invalidate the STag and continue to reuse it later (608).

Both fast memory registration and invalidation operations are defined as asynchronous operations. These are posted to the RNIC send queue as work requests, and their completion is reported through the associated completion queue.

RDMA defines two types of receive queues (RQs): shared and not shared receive queues (RQs). The shared RQ can be shared between multiple connections, and the receive WR posted to this queue can be consumed by outgoing messages received over different connections. A nonshared RQ is always associated with one connection, and the WR posted to this RQ is consumed by transmissions received over this connection.

Referring now to FIGS. 7 and 8, an offload of an iSCSI data movement operation by RDMA supporting RNIC will be described, in accordance with an embodiment of the present invention.

First, reference is made to FIG. 7. According to a non-limiting embodiment of the present invention, the conventional RDMA offload function can be divided into two parts, the RDMA service unit 700 and the RDMA messaging unit 701. The RDMA messaging unit 701 may process incoming and outgoing RDMA messages and may perform direct deployment and forwarding operations using the services provided by the RDMA service unit 700. To enable iSCSI offload, the iSCSI offload function may be replaced by and performed by iSCSI messaging unit 702. The iSCSI messaging unit 702 may be responsible for handling incoming and outgoing iSCSI PDUs and may also perform direct deployment and delivery using the services provided by the RDMA service unit 700.

The services and interfaces provided by RDMA service unit 700 are the same for both iSCSI and RDMA offload functionality.

Reference is now made to FIG. 8. All iSCSI PDUs are generated in software except data-out 802, which is generated in hardware (reference number 801). The generated iSCSI PDU may be posted to the transmit queue as a transmit work request (803). The RNIC reports 804 the completion (successful transmission operation) of these WRs through the associated completion queue.

The software is responsible for posting the buffer to a receive queue (e.g., having a receive job request) (805). Note that the receive buffer can generally be posted before the send buffer to avoid bad race conditions. The particular order of posting the transmit and receive buffers is not critical to the invention and may be left to the implementer. The buffer may be used for incoming control and unsolicited data-out PDUs (806). The RNIC may be extended (807) to support two RQs, one for the incoming iSCSI control PDU and the other for unsolicited incoming data-out. Software may use shared RQ to improve the utilization of buffers used for memory management and iSCSI control PDUs (808).

Control-received or unsolicited data-out PDUs may be reported using the completion queue (809). Data corruption or other errors detected in the iSCSI PDU data may be reported 810 via a completion queue for the iSCSI PDU consuming WQE in the RQ or through an asynchronous event queue for the data moving iSCSI PDU. The RNIC may then process 811 the PDU.

According to a non-limiting embodiment of the present invention, implementation of iSCSI semantics using an RDMA-based mechanism can be implemented with a unified software architecture for iSCSI and iSER-based solutions.

Referring now to FIG. 9, a software architecture implemented using RDMA-based iSCSI offload is described. SCSI layer 900 communicates with iSCSI driver 901 via an iSCSI application protocol. The data mover interface 902 interfaces with the iSCSI driver 901, the iSER data mover 903, and the iSCSI data mover 904. The manner in which the data mover interface 902 interfaces with these components may be in accordance with the standard data mover interface defined by the RDMA consortium. One non-limiting advantage of this software architecture is the high sharing of software components and interfaces between the iSCSI and iSER software stacks. The data mover interface allows the iSCSI driver to split data movement and iSCSI management functions. In simple terms, the data mover interface is used when the SCSI layer 900 requests the transmission of a command, for example to complete a SCSI command to an initiator, or, for example, to complete a portion of a SCSI command to a target. This ensures that all necessary data transmissions occur when requesting the transmission / reception of a sequence.

The functionality of the iSCSI and iSER data movers 903 and 904 may be offloaded to the RDMA-based service 905 implemented by the RNIC 906. According to one embodiment of the invention, offloading the iSCSI function using the RDMA mechanism includes offloading both the iSCSI target and the iSCSI initiator function. Each offload function (target and / or initiator) may be implemented separately from and independent of other functions or end-points. In other words, the initiator may have an offloaded data movement operation and still communicate with any other iSCSI implementation of the target without requiring any changes or modifications. The same applies to the offloaded iSCSI target function. All RDMA mechanisms used to offload the iSCSI data movement feature are local and transparent to the remote side.

Referring now to FIG. 10, direct data placement of an iSCSI data movement PDU to a SCSI buffer without hardware / software interaction is described, in accordance with an embodiment of the present invention. First, the RNIC is provided a description of the SCSI buffer (eg, by software) (reference number 1001). Each SCSI buffer may be uniquely identified by ITT or TTT, respectively (1002). The SCSI buffer may consist of one or more pages or blocks and may be represented as a page-list or a block-list.

To perform direct data placement, the RNIC may perform a two stage resolution process. The first step 1003 involves identifying a SCSI buffer given an ITT (or TTT), and the second step 1004 finds a page / block in the list for reading / writing to this page / block. It includes. Both the first and second steps can use the address translation and protection mechanisms defined by RDMA and can use the STag and RDMA memory registration semantics to implement iSCSI ITT and TTT semantics. For example, an RDMA protection mechanism can be used to locate a SCSI buffer and to protect the buffer from unsolicited access (1005), and the address translation mechanism can be used to page / block within a page-list or block-list. Efficient access may be enabled (1006). In order to perform RDMA-like remote memory access to the iSCSI data movement PDU, the initiator or target software may register the SCSI buffer (eg, using register memory area semantics) (1007). As a result of memory registration, a protection block is associated with the SCSI buffer. As such, the protection block points to a translation table entry that holds a page-list or block-list describing a SCSI buffer. The registered memory area may be a zero-based type of memory area that enables the use of a buffer offset in an iSCSI data movement PDU to access a SCSI buffer.

The ITT and TTT used in the iSCSI control PDU may get the value of STag referring to the registered SCSI buffer (1008). For example, a SCSI read command generated by the initiator may carry an ITT such as a STag of a registered SCSI buffer. Corresponding data-in and SCSI response PDUs may also carry this STag. Thus, the STag can be used to perform remote direct data placement by the initiator. In the case of a SCSI write command, the target may register its SCSI buffer allocated for the requested incoming data-out PDU, and may use a TTT such as the STag of the SCSI buffer in the R2T PDU (1009).

This non-limiting method of the present invention makes it possible to use existing hardware and software mechanisms to perform efficient offloading of iSCSI data movement operations while preserving the flexibility of those operations defined in the iSCSI specification.

Referring now to FIGS. 11A and 11B, in accordance with an embodiment of the present invention, processing data-in and requested data-out by the RNIC using the RDMA protection and address translation method described with reference to FIG. 10. And performing direct data placement of the iSCSI payload carried by the PDU into the registered SCSI buffer. In addition, the RNIC can track data sequencing of data-in and data-out, enforce iSCSI sequencing rules defined by the iSCSI specification, and perform PB invalidation at the end of data transactions.

Incoming data-in and requested data-out can be handled very similarly by the RNIC (by initiator and target, respectively). Processing common to both of these PDU types will now be described.

The RNIC first detects iSCSI data-in and requested data-out PDUs (1101). This can be accomplished by the BHS: Opcode and BHS: TTT fields, but is not limited to this (TTT = h'FFFFFFFF 'indicates that no data-out PDU is requested, and this PDU is a control iSCSI PDU as described above. Treated as). The RNIC can use the BHS: TTT field for data-in PDUs and BHS: TTT for data-out PDUs as STags (which were previously used by the driver when the driver individually created SCSI commands or R2Ts). have.

The RNIC may find the PB using, for example, an index field of the STag that describes the individual registered SCSI buffers and verifies access permissions (1102). The RNIC may know where the data is accessed within a registered SCSI register, for example using BHS: BufferOffset (1103). The RNIC may then use the address translation mechanism to break up the page / block and perform direct data placement (or direct data read) to the registered SCSI buffer (1104).

Consumer software (drivers) do not know the direct deployment operations performed by the RNIC. There is no completion notification except in the case of the requested data-out PDU with the 'F-bit' set.

In addition to (eg, prior to) a direct placement operation, the RNIC may perform sequence verification of the incoming PDU (1105). Both data-in and data-out PDUs carry a DataSN. The DataSN may be zero for each SCSI command in case of data-in and zero for each R2T in case of data-out (1106). The RNIC may keep the ExpDataSN in a protected block (1107). This field may be initialized to zero at PB initialization (fast memory registration) 1108. For each incoming data-in or requested data-out PDU, this field may be compared to BHS: DataSN (1109).

a. If DataSN = ExpDataSN, the PDU is accepted and processed by the RNIC, and the ExpDataSN is incremented (1110).

b. If DataSN> ExpDataSN, an error is reported 1111 (related asynchronous error-sequencing error) to the software, such as by using an asynchronous event notification mechanism. The ErrorBit in the PB can then be set, and each incoming PDU that refers to this PB (using a stag) is discarded starting from this point. This means that the iSCSI driver needs to recover at the iSCSI command level (or at the R2T level respectively).

c. The last case is the reception of ghost PDUs (DataSN <ExpDataSN). In that case, the received PDU is discarded and no error is reported to software (1112). This makes it possible to handle replicated iSCSI PDUs as defined by the iSCSI specification.

In the case of a SCSI read command, the initiator receives 1113 one or more data-in PDUs followed by a SCSI response. The SCSI response can carry BHS: ExpDataSN. This field indicates the number of data-ins before the SCSI response. To complete enforcement of the iSCSI sequencing rules, the RNIC may compare BHS: ExpDataSN with PB: ExpDataSN referred to by the STag (ITT) carried by its SCSI response. In the case of a mismatch, a completion error is reported, indicating that a sequencing error has been detected (1114).

The requested data-out PDU with the 'F-bits' set indicates that this PDU completes the transaction requested by the corresponding R2T (1115). In that case, the completion notification is sent to the consumer software (1116). For example, the RNIC may skip one WQE from the receive queue and add a CQE to its completion queue to indicate the completion of the data-out transaction. The target software may need this notification to know whether the R2T operation has completed and whether R2T can generate a SCSI response confirming that the entire SCSI write operation has completed. Note that this notification may be the only notification from the RNIC to the software when processing the incoming data-in and requested data-out PDUs. The sequencing verification described above ensures that all data-outs have been successfully received and placed in a registered buffer. The loss of the last data-out PDU (delivering the set 'F-bits') can be handled by software (timeout mechanism).

The last operation that may be performed by the RNIC to finish processing the data-in and requested data-out PDUs is invalidation of the protection block (1117). This may be done for the requested data-out PDU with the data-in and 'F bits' set. Invalidation may be performed for the PB referred to by the STag collected from the PDU header. The invalidated STag can be passed to the SCSI driver using CQE for the requested data-out or as a header (ITT field) of the SCSI response to complete the SCSI write command. This allows the iSCSI driver to reuse the freed STag for the next SCSI command.

Invalidation 1118 of the area registered by the target may similarly be performed. Note that an alternative method of invalidation may be invalidation of the PB referred to by the STag (ITT) in the received SCSI response.

Referring now to FIG. 12, the processing of an incoming R2T in hardware and the generation of a data-out PDU, in accordance with one embodiment of the present invention, are described.

The result initiator of the SCSI write command may receive a number of R2Ts from the target (1201). Each R2T may require the initiator to fetch a specified amount of data from a designated location in a registered SCSI buffer and send this data to the target using a data-out PDU (1202). R2T passes the ITT provided by the initiator in a SCSI command (1203). As described above, the STag of the registered SCSI buffer can be used by the driver instead of the ITT (1204) when the driver generates a SCSI command.

R2T PDUs may be identified using the BHS: Opcode field. The RNIC may perform verification of R2T sequencing using the BHS: R2TSN field (1205). The RNIC holds the ExpDataSN field in the PB. For unidirectional commands, the same field can be used for sequencing verification because the initiator can see the R2T or data-in coming in. The sequence verification for the incoming R2T may be the same as the process of sequence verification used for data-in and data-out described above (1206).

The RNIC may use the same mechanism as it handles the incoming RDMA read request to process the R2T that passed the sequence verification (1207). The RNIC may use a separate read response work queue to post a WQE that describes the data-outs that need to be sent by the RNIC transmit logic (1208) (in the case of an RDMA read request, the RNIC reads an RDMA read response). Queue the WQE that describes The transmission logic may arbitrate between the sending WQ and the read response WQ and may process WQE from each of them according to an internal arbitration rule (1209).

A single data-out PDU may be obtained 1210 as a result of each received R2T. The generated data-out PDU may carry data from the registered SCSI buffer referred to by BHS: ITT (the driver places the STag there upon SCSI command generation). BHS: BufferOffset and BHS: DesireDataTransferLength can identify the offset in the SCSI buffer and the size of the data transaction.

When the RNIC sends a data-out for an R2T PDU with the F-bit set, the RNIC may invalidate the protection block mentioned by the STag (ITT) after the remote side confirms successful receipt of that data-out PDU. . The STag used for this SCSI write command can be reused by software when the corresponding SCSI response PDU is delivered.

An alternative method for memory area invalidation may be invalidation of the PB mentioned by the STag (ITT) in the received SCSI response.

The description of the invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments have been chosen and described in order to best explain the principles, practical applications of the invention and to enable those skilled in the art to understand the invention with respect to various embodiments having various modifications suitable for the particular use contemplated.

Claims (28)

Internet Small Computer System Interface (SCSI) offload initiator capability using the Remote-direct-memory-access-enabled Network Interface Controller (RNIC) mechanism used for the Remote Direct Memory Access (RDMA) feature. providing a function); Applying the RDMA function to use a regular TCP connection and to map iSCSI directly through a transport control protocol (TCP) layer How to implement the iSCSI offload (offload) comprising a. The method of claim 1, offloading the iSCSI initiator function independently from the iSCSI target function How to implement the iSCSI offload (offload) further comprising. The method of claim 1, The step of providing the iSCSI offload initiator function, Remote direct data placement of data-out payloads in pre-registered SCSI buffers in any order according to any SCSI buffer offset using the logic of the RDMA write operation. To include, How to implement iSCSI offload. The method of claim 3, Identifying the pre-registered SCSI buffers by a target task tag (TTT) used as a steering tag (STag) How to implement the iSCSI offload (offload) further comprising. The method of claim 1, Providing the iSCSI offload initiator function, Deploying control iSCSI PDUs using RDMA receive queues with Receive Work Requests To include, How to implement iSCSI offload. The method of claim 5, Reporting the completion of the incoming work requests through an associated completion queue How to implement the iSCSI offload (offload) further comprising. The method of claim 1, Providing the iSCSI offload initiator function, providing a SCSI layer to communicate with the iSCSI driver through the iSCSI application protocol, Providing an iSER (iSCSI Extensions for RDMA) data mover and an data mover interface to interface with the iSCSI data mover; To include, How to implement iSCSI offload. The method of claim 7, wherein Dividing the data movement function and the iSCSI management function of the iSCSI driver by using the data mover interface How to implement the iSCSI offload (offload) further comprising. delete delete A computer-readable recording medium implementing iSCSI offload, which records a program comprising instructions for executing each step according to any one of claims 1 to 8. delete delete delete delete delete delete delete delete A system for implementing iSCSI offload, comprising means for executing each step according to any one of claims 1 to 8. delete delete delete delete delete delete delete delete

Images