JP7553783B2 - STORAGE CONTROL DEVICE, DELIVERY STATUS DETECTION PROGRAM, AND STORAGE SYSTEM - Google Patents

Description

The present invention relates to a storage control device, a delivery status determination program, and a storage system.

In recent years, storage systems that use NVMe (Non-Volatile Memory Express) SSDs (Solid State Drives) as storage devices have been developed. NVMe is a communications protocol for optimizing communications in flash storage that uses non-volatile memory.

In a prior art storage device, a controller creates a queue group in the controller itself or in a storage device, each containing multiple command queues with different priorities, and posts commands to the storage device that require faster processing to a command queue with a higher priority. There is also a technology in which a DMA (Direct Memory Access) controller reads the transfer status of data transferred to another CM (Controller Module) via a switch from the switch, and writes the read transfer status to memory.

International Publication No. 2017/158799 JP 2015-18314 A

However, in conventional technology, when accessing a storage device such as an NVMe SSD via another storage control device, there is a problem in that it is not possible to ensure delivery of data transferred from the storage device.

In one aspect, the present invention aims to make it possible to determine the delivery status of data.

In one embodiment, a storage control device is provided that includes an integrated circuit or dedicated integrated circuit with reconfigurable internal circuits, which includes a buffer area for data transfer and issues an interrupt specifying an identifier corresponding to the buffer area when writing is performed to the buffer area, and has a control unit that, when accessing a storage device that performs data transfer without a response via another storage control device, requests the transfer of read data to its own memory and transmits a read command execution request to the other storage control device that requests the transfer of specified data to a buffer area corresponding to a specified first identifier in the integrated circuit, enables an interrupt flag corresponding to the first identifier when an interrupt specifying the first identifier is received from the integrated circuit, and receives a completion response specifying the first identifier from the other storage control device when execution of the read command issued from the other storage control device to the storage device in response to the execution request is completed, and, when the completion response is received, determines the delivery status of the read data by referring to the interrupt flag corresponding to the first identifier.

One aspect of the present invention has the effect of making it possible to determine the delivery status of data.

FIG. 1 is an explanatory diagram illustrating an example of a system configuration of a storage system according to an embodiment. FIG. 2 is an explanatory diagram showing one embodiment of a storage system 100. FIG. 3 is an explanatory diagram showing an example of the data structure of an NVMe command. FIG. 4 is an explanatory diagram showing an example of the contents stored in the ID correspondence table. FIG. 5 is a block diagram showing an example of the functional configuration of the CM in charge #i. FIG. 6 is a block diagram showing an example of the functional configuration of DCM#i. FIG. 7 is an explanatory diagram showing an example of the operation of the storage system 100. FIG. 8 is a sequence diagram showing an example of a processing procedure of the storage system 100 when the storage system 100 is operating normally. FIG. 9 is a diagram (part 1) showing an example of updating the ID correspondence table 400. In FIG. FIG. 10 is a sequence diagram showing an example of a processing procedure when an abnormality occurs in the storage system 100. In FIG. FIG. 11 is a diagram (part 2) showing an example of updating the ID correspondence table 400. In FIG. FIG. 12 is an explanatory diagram (part 1) showing another embodiment of the storage system 100. In FIG. FIG. 13 is an explanatory diagram (part 2) showing another embodiment of the storage system 100. In FIG.

The following describes in detail the embodiments of the storage control device, delivery status determination program, and storage system according to the present invention with reference to the drawings.

(Embodiment)
Fig. 1 is an explanatory diagram showing an example of a system configuration of a storage system according to an embodiment. In Fig. 1, a storage system 100 includes CM #0, CM #1, and a storage device $0. CM #0, #1, and storage device $0 are provided in a CE (Controller Enclosure) $0. The CE is a case (box) for housing the CMs and the like.

CM#0 and #1 are examples of storage control devices that control access to storage device $0 under their control. CM#0 and #1 in the same CE$0 are directly connected so that they can communicate with each other. CE$0 is realized, for example, as a single storage device.

Storage device $0 is a storage device that performs data transfers that do not require a response. For example, storage device $0 is an NVMe SSD. NVMe uses PCIe (Peripheral Component Interconnect-Express) as the connection standard for SSDs. PCIe uses posted writes, so write transfers that do not require a response are performed.

In the following explanation, "NVMe SSD $0" will be used as an example of storage device $0. NVMe SSD $0 is an SSD that uses the NVMe communication protocol. However, other storage devices that perform data transfers that do not require a response may also be used as storage device $0.

A host device 110 is connected to CM#0 and #1. The host device 110 is, for example, a server that executes business applications. CM#0 and #1 and the host device 110 are connected, for example, via a SAN (Storage Area Network) that uses FC (Fibre Channel) or iSCSI (Internet Small Computer System Interface).

In the following description, any one of CMs #0 and #1 in storage system 100 may be referred to as "CM #i" (i = 0, 1). Also, CM #i that has accepted a request (I/O request) from host device 110 may be referred to as the "responsible CM."

CM#i includes, for example, a CPU (Central Processing Unit)#i, a memory#i, a CA (Channel Adapter)#i, an FPGA (Field Programmable Gate Array)#i, and a PCIe SW (switch)#i. Each component is connected to the other components via a bus.

CPU #i is responsible for the overall control of CM #i. CPU #i may have multiple cores. Memory #i stores data and programs. Memory #i includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), and flash ROM. Specifically, for example, flash ROM and ROM store various programs, and RAM is used as a work area for CPU #i. Programs stored in memory #i are loaded into CPU #i to cause CPU #i to execute the coded processes. Memory #i may also be used as a cache memory that temporarily stores data and programs.

CA#i is an interface for communicating with the host device 110. CA#i has an adapter that is compliant with, for example, LAN, SAN, FC, etc. FPGA#i is a programmable logic device, an integrated circuit whose internal circuitry can be reconfigured. PCIe SW#i is an interface for communicating with other devices (CM, storage device). CM#i may have, for example, an IOC (Input/Output Controller) that controls access to NVMe SSD$0.

The number of CEs included in storage system 100 is not limited to one, and the number of CMs included in a CE is not limited to two. For example, storage system 100 may include multiple CEs (see, for example, FIG. 12 described below).

In addition, in FIG. 1, a case where one host device 110 is connected to CM#0 and #1 has been described as an example, but this is not limited to this. For example, each of multiple host devices may be connected to one or more of CM#0 and #1.

In the example of FIG. 1, CM#i has FPGA#i, but this is not limited to the above. For example, the processing performed by FPGA#i in CM#i may be realized by a dedicated integrated circuit (e.g., LSI: Large Scale Integration). In other words, CM#i may have a dedicated integrated circuit with the same functions as FPGA#i instead of FPGA#i.

Here, in the case of a SAS (Serial Attached SCSI) SSD, the CM in charge has a path from the IOC of the own system to the EXP of the other system, so the SSD can be accessed from both the EXP of the own system and the EXP of the other system. The EXP (expander) is a SAS EXP, and is a relay device for connecting the controller to multiple SAS disk drives.

On the other hand, when NVMe SSDs are used as storage devices instead of SAS SSDs, if the SAS EXP is replaced with a PCIe SW, the path from the IOC of the own system to the EXP of the other system is lost, so the responsible CM can only access one of the SSDs. Therefore, for example, to exchange data with an NVMe SSD under the control of another CM, access must be made via the other CM. In this case, for example, if access is made via the buffer of the other CM, the response is slow, so it is desirable to exchange data (read data) directly between the responsible CM and the SSD.

However, in conventional storage systems that use NVMe SSDs as storage devices, there is a problem in that data delivery cannot be guaranteed when accessing an NVMe SSD in a different PCIe domain. For example, consider a case where a failure occurs between the PCIe SW of the CM in charge and the NVMe SSD.

In this case, the responsible CM requests the other CM to access the NVMe SSD. The other CM is a CM that can access the SSD in which the data that the responsible CM wants to read is stored. In the following explanation, a CM that can access the SSD in which the data that the responsible CM wants to read is stored may be referred to as a "DCM (Drive CM)."

In response to a request from the CM in charge, the DCM issues a read command to the NVMe SSD. In response to the read command from the DCM, the NVMe SSD DMA transfers the read data to the memory of the CM in charge. At this time, due to PCIe posted write, delivery of the read data transferred to the memory of the CM in charge cannot be guaranteed.

In other words, even if a failure occurs such as a temporary disconnection of the link between PCIe SWs, it is not possible to determine whether the transfer of read data from the NVMe SSD to the memory of the assigned CM was successful. If data delivery cannot be guaranteed, data loss cannot be detected, which may cause data corruption in the host device.

Specifically, for example, when the NVMe SSD completes the transfer process of the read data to the memory of the assigned CM, it issues an MSI (Message Signaled Interrupts)-X interrupt to the DCM. When the DCM determines that the transfer process of the read data is complete based on the MSI-X interrupt from the NVMe SSD, it sends a command completion response to the assigned CM.

However, even if a completion response is sent from the DCM to the responsible CM, the responsible CM cannot determine whether the data transfer was successful or not. In other words, since there is no response from the memory of the responsible CM accompanying the data transfer (posted write), the responsible CM cannot recognize if the data was not transferred successfully. Even in this case, a completion response is sent from the DCM to the responsible CM, and garbled data occurs in the host device.

In this embodiment, a storage system 100 that can ensure data delivery guarantees even when CM#i (responsible CM) accesses NVMe SSD$k via another CM#j (DCM) is described. Specifically, for example, CM#i (responsible CM) is provided with an FPGA#i that includes a buffer area for data transfer and issues an interrupt specifying an identifier corresponding to the buffer area when writing is performed to the buffer area. Then, CM#i (responsible CM) determines the delivery status of read data to memory #i from the writing status to the buffer area in FPGA#i, which is performed using the same path as the data transfer from NVMe SSD$k to memory #i.

Figure 2 is an explanatory diagram showing one embodiment of the storage system 100. Here, CM#0 is the responsible CM, and CM#1 is the DCM. Also, assume that a failure occurs between PCIe SW#0 of responsible CM#0 and NVMe SSD$0. In this case, responsible CM#0 accesses NVMe SSD$0 via DCM#1. Note that Figure 2 shows an excerpt of a portion of the hardware configuration of each of CM#0 and #1.

The responsible CM#0 provides a doorbell register (DBL0, 1, ..., N) corresponding to the SQ ID in FPGA#0. The doorbell register (DBR) is an example of a buffer area for data transfer. The SQ ID is an identifier that uniquely identifies an SQ (Submission Queue) described below. The responsible CM#0 maps the doorbell register area of FPGA#0 to memory#0. For example, doorbell0 in memory#0 corresponds to DBL0 in FPGA#0.

In addition, the responsible CM#0 provides MSI-X0, 1, ..., N in memory #0 that set interrupts from FPGA#0. When a write is performed to the doorbell register, FPGA#0 issues an MSI-X interrupt corresponding to the ID of the doorbell register to CPU#0 of CM#0.

DCM#1 creates N pairs of SQ and CQ (Completion Queue) in memory#1. N can be set arbitrarily, for example, to a value corresponding to the number of cores of CPU#1. SQ and CQ are used to access NVMe SSD$0 from CPU#1 of DCM#1 for each NVMe command.

An SQ is a queue that stores commands. A CQ is a queue that stores information indicating the completion status of a command. Specifically, for example, DCM #1 creates a pair of an SQ and a CQ in memory #1 for each SQ ID (0, 1, ..., N). For example, "SQ 0" and "CQ 0" in memory #1 indicate the pair of an SQ and a CQ corresponding to SQ ID "0".

Here we explain an example of the data structure of an NVMe command.

Figure 3 is an explanatory diagram showing an example of the data structure of an NVMe command. In Figure 3, the NVMe command cmd includes an Opcode, a data address 1, a data address 2, a Starting LBA (Logical Block Addressing), and a Number of Logical Blocks.

Here, Opcode indicates the type of command. For example, in the case of a read command, Opcode is Read. Data address 1 indicates the destination address of the data area in the memory of the responsible CM. Data address 2 indicates the address of the Doorbell register in the FPGA of the responsible CM.

Starting LBA indicates the first address of the block to be read from the NVMe SSD. Number of Logical Blocks indicates the number of blocks to be read from the NVMe SSD. One block is, for example, 512 bytes. For example, the number of blocks of data to be transferred to the data area in the memory of the responsible CM is L blocks.

In this case, the Number of Logical Blocks is, for example, (L + α) blocks in order to add data transfer to the Doorbell register. α blocks are assigned to the transfer to the Doorbell register. α can be set arbitrarily, for example, to 1.

In the storage system 100, the responsible CM #0 uses an ID correspondence table 400 as shown in FIG. 4 to associate the SQ and CQ of DCM #1, the doorbell register of FPGA #0, and the ID of MSI-X. The ID correspondence table 400 is stored, for example, in memory #0 of the responsible CM #0.

Figure 4 is an explanatory diagram showing an example of the contents stored in an ID correspondence table. In Figure 4, ID correspondence table 400 has fields for command ID, SQ ID, CQ ID, Doorbell ID, MSI-X ID, and interrupt flag. By setting information in each field, ID correspondence information (for example, ID correspondence information 400-1) is stored as a record.

The command ID is an identifier that uniquely identifies a command. The SQ ID is an identifier that uniquely identifies an SQ in memory #1 of DCM #1. The CQ ID is an identifier that uniquely identifies a CQ in memory #1 of DCM #1. The Doorbell ID is an identifier that uniquely identifies a Doorbell register in FPGA #0 of the responsible CM #0.

The MSI-X ID is an identifier that uniquely identifies an MSI-X interrupt from FPGA#0. Each ID is a value between 0 and N. N corresponds to the number of SQ and CQ pairs created in the DCM memory. The interrupt flag indicates the status of the MSI-X interrupt from FPGA#0. An interrupt flag of "0" indicates that no interrupt has occurred (disabled). An interrupt flag of "1" indicates that an interrupt has occurred (enabled). In the initial state, the interrupt flag is "0".

In storage system 100, the same ID is used for the SQ, CQ, Doorbell register, and MSI-X interrupt in the sequence of events related to each read command. This allows CPU#0 of responsible CM#0 to determine which SQ ID command has been completed (delivery status) by checking the ID of the MSI-X interrupt received from FPGA#0.

(Example of functional configuration of CM#i)
Next, a functional configuration example of CM#i will be described. First, a functional configuration example in the case where CM#i operates as a responsible CM will be described with reference to FIG. 5. Here, the DCM in the case where CM#i is the responsible CM may be written as "DCM#j" (j≠i). For example, when CM#0 is the responsible CM (i=0), CM#1 becomes the DCM (j=1). In addition, the NVMe SSD of the access destination having the read data may be written as "NVMe SSD$k".

Figure 5 is a block diagram showing an example of the functional configuration of the responsible CM #i. In Figure 5, the responsible CM #i includes a first reception unit 501, a first request processing unit 502, a judgment unit 503, and a recovery unit 504. The first reception unit 501 to the recovery unit 504 are functions that become a control unit, and specifically, for example, the functions are realized by having the CPU #i execute a program stored in the memory #i, or by the CA #i and PCIe SW #i. The processing results of each functional unit are stored in the memory #i, for example.

The first reception unit 501 receives a request from the host device 110. Here, the request is, for example, a read request (read request) for the NVMe SSD$k. The request includes, for example, information on the access destination and the data size of the read data. Specifically, for example, the first reception unit 501 receives a read request from the host device 110 via CA#i.

When the first request processing unit 502 accesses the NVMe SSD$k via DCM#j (another storage control device), it sends to the DCM#j a request to execute a read command that requests the NVMe SSD$k to transfer read data to its own memory #i, specifying the identifier (first identifier) that corresponds to the read command.

Here, the NVMe SSD$k is accessed via DCM#j when, for example, a failure occurs between the PCIe SW#i of the responsible CM#i and the NVMe SSD$k. Also, the NVMe SSD$k is accessed via DCM#j when the NVMe SSD$k is located in a CE different from the responsible CM#i.

The read command is, for example, the NVMe command cmd shown in FIG. 3. The read data is L blocks of data read from the Starting LBA of the NVMe SSD $k, and is transferred to the data area in the memory #i of the responsible CM #i.

The first identifier corresponding to a read command is, for example, an SQ ID assigned to the read command. For example, each read command is assigned an available SQ ID with a smaller value among SQ IDs 0 to N. An available ID is an ID that does not correspond to other read commands, for example, an ID that is not registered in the ID correspondence table 400 shown in FIG. 4.

When a request to execute a read command specifying the first identifier is sent to DCM#j, a read command is issued from DCM#j to NVMe SSD$k, to which a transfer request for specific data to the buffer area corresponding to the first identifier in FPGA#i is added. The buffer area corresponding to the first identifier is, for example, a Doorbell register corresponding to the SQ ID.

In other words, a request to execute a read command specifying a first identifier corresponds to a request to execute a read command that requests the transfer of read data to the local memory #i and also requests the transfer of specified data to the buffer area corresponding to the first identifier in the local FPGA #i.

The predetermined data can be set arbitrarily, and is, for example, data having a smaller amount of data than the read data requested by the host device 110. The predetermined data may be one block of data following the read data. In addition, the predetermined data is requested to be transferred to a buffer area corresponding to a first identifier in FPGA #i, for example, after the transfer of the read data from NVMe SSD $k to memory #i is completed. The specific processing content on the DCM side will be described later with reference to FIG. 6.

Specifically, for example, when the first request processing unit 502 accesses the NVMe SSD$k via DCM#j in response to a read request from the host device 110, the first request processing unit 502 refers to the ID correspondence table 400 to identify the SQ ID to be assigned to the read command. For example, if the ID correspondence information is not registered in the ID correspondence table 400, SQ0 (ID:0) is identified.

The first request processing unit 502 creates ID correspondence information (for example, ID correspondence information 400-1 shown in FIG. 4) corresponding to the identified SQ0 and registers it in the ID correspondence table 400. Then, the first request processing unit 502 specifies the identified SQ0 (ID:0) and sends a request to DCM#j to execute a read command for NVMe SSD$k. Information on the access destination and the data size of the read data are identified from the read request.

When the first request processing unit 502 receives an interrupt specifying a first identifier issued from FPGA#i, the first request processing unit 502 enables an interrupt flag corresponding to the first identifier. An interrupt specifying the first identifier is issued from FPGA#i when writing is performed to a buffer area corresponding to the first identifier provided in FPGA#i.

Specifically, for example, when a write is performed to the Doorbell register of Doorbell0 (ID:0), FPGA#i issues an MSI-X interrupt specifying Doorbell0 (ID:0) to CPU#0. The MSI-X interrupt specifying Doorbell0 (ID:0) is the MSI-X interrupt of MSI-X0 (ID:0) associated with Doorbell0 (ID:0). In this case, the first request processing unit 502 refers to the ID correspondence table 400 and sets the interrupt flag of the ID correspondence information 400-1 corresponding to MSI-X0 (ID:0) to "1".

When execution of a read command issued from DCM#j to NVMe SSD$k in response to an execution request sent by the first request processing unit 502 is completed, the determination unit 503 receives a completion response specifying the first identifier from DCM#j.

The issued read command is a read command that requests the transfer of read data to memory #i, and also requests the transfer of specified data to a buffer area corresponding to a first identifier in FPGA #i. The request to transfer specified data to a buffer area corresponding to a first identifier in FPGA #i is added, for example, in DCM #j by manipulating a read command that requests the transfer of read data to memory #i.

When the determination unit 503 receives a completion response specifying a first identifier, the determination unit 503 refers to the interrupt flag corresponding to the first identifier to determine the delivery status of the read data. Specifically, for example, when the determination unit 503 receives a completion response specifying an SQ ID from DCM#j, the determination unit 503 refers to the ID correspondence table 400 to check the value of the interrupt flag corresponding to the specified SQ ID.

Here, if the interrupt flag is "1 (enabled)", the determination unit 503 determines that the read data has been delivered. In other words, the determination unit 503 determines that the transfer of the read data to memory #i has been successful. The read data is, for example, L blocks of data requested by the host device 110.

On the other hand, if the interrupt flag is "0 (invalid)", the determination unit 503 determines that the read data was not delivered. In other words, the determination unit 503 determines that the transfer of the read data to memory #i was not performed normally. Note that the determination unit 503 also determines that the read data was not delivered when, for example, an error response specifying an SQ ID is received from DCM #j.

When it is determined that the read data has not been delivered, the recovery unit 504 executes a predetermined recovery process. The predetermined recovery process is, for example, a process of reconstructing the path (link) between the memory #i and the NVMe SSD $k. In other words, the recovery process is executed because there is a possibility that a temporary hardware failure has occurred, causing the link to be unstable or the communication speed of the link not being at the specified speed.

In addition, when the specified recovery process is completed, the first request processing unit 502 may again send to DCM #j a request to execute a read command requesting NVMe SSD $k to transfer read data to its own memory #i, specifying the first identifier corresponding to the read command.

If the read data is not delivered even when the read command execution request is resubmitted after the recovery process is executed, the recovery unit 504 may notify the host device 110 of this fact. In addition, the read data that was successfully transferred to memory #i is transferred to the host device 110.

Next, an example of a functional configuration when CM#i operates as a DCM will be described with reference to FIG. 6. Here, when CM#i is a DCM, the responsible CM may be written as "responsible CM#j" (j ≠ i). Also, an NVMe SSD under DCM#i may be written as "NVMe SSD$k".

Figure 6 is a block diagram showing an example of the functional configuration of DCM#i. In Figure 6, DCM#i includes a second reception unit 601 and a second request processing unit 602. The second reception unit 601 and the second request processing unit 602 are functions that become a control unit, and specifically, for example, the functions are realized by having the CPU#i execute a program stored in the memory#i, or by the CA#i and PCIe SW#i. The processing results of each functional unit are stored in the memory#i, for example.

The second reception unit 601 receives a request to execute a read command from the responsible CM #j (another storage control device) that specifies an identifier (second identifier) corresponding to the read command that requests the NVMe SSD $k to transfer read data to the memory #j of the responsible CM #j. The specified second identifier is, for example, an SQ ID assigned to the read command.

When the second request processing unit 602 receives a request to execute a read command specifying a second identifier, it requests the transfer of read data to the other memory #j and issues a read command to the NVMe SSD $k requesting the transfer of specified data to a buffer area corresponding to the specified second identifier in the other FPGA #j.

In other words, a request to execute a read command specifying the second identifier corresponds to a request to execute a read command that requests the transfer of read data to the other memory #j and also requests the transfer of specified data to a buffer area in the other FPGA #j that corresponds to the second identifier. The address of the buffer area in the other FPGA #j that corresponds to the second identifier may be included in the execution request from the responsible CM #j, for example, or may be queried from the responsible CM #j.

When the second request processing unit 602 receives a transfer completion notification from the NVMe SSD $k of the data requested by the read command, it sends a completion response specifying the second identifier to the responsible CM #j. The completion response indicates that the read command has been completed. In addition, when the second request processing unit 602 receives a transfer error notification from the NVMe SSD $k of the data requested by the read command, it sends an error response specifying the second identifier to the responsible CM #j. The error response indicates that the read command did not complete successfully.

Specifically, for example, the second request processing unit 602 adds a request to transfer specific data to a buffer area corresponding to the second identifier in the other FPGA #j to a read command requesting the transfer of read data to the other memory #j. Then, the second request processing unit 602 stores the read command to which the transfer request has been added in a first queue corresponding to the second identifier in the own memory #i. The first queue is a queue that stores commands, and is, for example, an SQ.

Next, when information indicating the completion of the read command is written by the NVMe SSD $k to the second queue corresponding to the second identifier in the own memory #i, the second request processing unit 602 transmits a transfer completion response specifying the second identifier to the responsible CM #j. The second queue is a queue that stores information indicating the completion status of the command, and is, for example, a CQ.

A more detailed example of the operation when accessing the NVMe SSD$k will be described later with reference to, for example, Figures 8 and 10.

(Operation Example of Storage System 100)
Next, an example of the operation of the storage system 100 will be described with reference to Fig. 7. Here, CM#0 is the responsible CM, and CM#1 is the DCM. Also, assume that a failure occurs between PCIe SW#0 of the responsible CM#0 and NVMe SSD$0, and the responsible CM#0 cannot directly access NVMe SSD$0.

Figure 7 is an explanatory diagram showing an example of the operation of storage system 100. In Figure 7, (1) each CM #0, #1 initializes the configuration devices and performs memory mapping of each memory #0, #1. (2) The responsible CM #0 specifies the SQ-ID and sends a request to execute a read command to DCM #1. The read command is a read command that requests NVMe SSD $0 to transfer read data to memory #0.

(3) DCM#1 adds a transfer request for specific data (one block of data) to the doorbell register corresponding to the SQ-ID in FPGA#0 of assigned CM#0 to the read command in the command format shown in FIG. 3. DCM#1 then sends the read command to NVMe SSD$0.

Specifically, for example, DCM#1 sets the destination address of the data area in memory#0 of assigned CM#0 to data address 1 of the read command. DCM#1 also sets the address of the Doorbell register corresponding to the SQ-ID in FPGA#0 of assigned CM#0 to data address 2. DCM#1 also sets the Number of Logical Blocks as "L+1".

(4) In response to the read command, NVMe SSD $0 writes L blocks of read data from the Starting LBA to the data area of memory #0 of assigned CM #0 (the address stored in data address 1 of the read command).

(5) NVMe SSD$0 writes one block of data following the read data to the doorbell register area (address stored in data address 2) in FPGA#0 of assigned CM#0.

(6) FPGA #0 issues an interrupt to CPU #0 of assigned CM #0, specifying the MSI-X ID associated with the Doorbell ID of the Doorbell register for which writing has been completed. For example, when writing to the Doorbell register for Doorbell ID "0" is completed, an interrupt specifying MSI-X ID "0" is issued to CPU #0. In this case, CPU #0 sets the interrupt flag corresponding to MSI-X ID "0" in ID correspondence table 400 to "1."

(7) When CPU #0 receives a transfer completion response specifying an SQ-ID from DCM #1, it refers to the interrupt flag corresponding to the specified SQ-ID in the ID correspondence table 400 to determine the delivery status of the read data.

This allows the responsible CM #0 to determine whether the transfer of read data was successful when accessing NVMe SSD $0 via DCM #1.

(Processing Procedure of Storage System 100)
Next, the processing procedure of the storage system 100 will be described. First, the processing procedure of the storage system 100 in a normal state will be described with reference to Fig. 8. However, CM#0 is the responsible CM, and CM#1 is the DCM. Also, assume that a failure occurs between PCIe SW#0 of the responsible CM#0 and NVMe SSD$0, and the responsible CM#0 cannot directly access NVMe SSD$0.

Figure 8 is a sequence diagram showing an example of a processing procedure when the storage system 100 is operating normally. Figure 9 is an explanatory diagram (part 1) showing an example of updating the ID correspondence table 400. In the sequence diagram of Figure 8, first, CPU #0 of responsible CM #0 creates ID correspondence information in the ID correspondence table 400 (step S801).

In this case, the SQ ID to be assigned to the newly issued command is "1". In this case, as shown in FIG. 9A, ID correspondence information 400-2 corresponding to SQ ID "1" is created and newly registered in ID correspondence table 400. Note that ID correspondence information 400-1 corresponding to SQ ID "0" assigned to the read command being requested for execution is registered in ID correspondence table 400.

Next, CPU#0 of responsible CM#0 specifies the identified SQ ID "1" and sends a request to DCM#j to execute a read command requesting NVMe SSD$k to transfer read data to memory#i (step S802).

When CPU #1 of DCM #1 receives a request to execute a read command specifying SQ ID "1", it writes the read command to which a request to transfer specified data to the Doorbell register corresponding to SQ ID "1" in FPGA #0 of assigned CM #0 is added, to the SQ corresponding to SQ ID "1" in memory #1 (step S803).

For example, a read command in the format shown in FIG. 3 is written to the SQ. Note that information specifying the Opcode, data address 1, data address 2, Starting LBA, and Number of Logical Blocks of the read command is included in the execution request from, for example, responsible CM #0. To explain in more detail, for example, CPU #1 of DCM #1 sets the destination address for the Doorbell register corresponding to SQ ID "1" to data address 2, and sets the Number of Logical Blocks as "L+1".

Next, CPU #1 of DCM #1 writes the SQ Tail value to the Submission Queue Doorbell register of NVMe SSD $0 (step S804). The Tail value is used to indicate to NVMe SSD $0 that a new command has been sent for processing.

Next, NVMe SSD$0 acquires a read command from SQ on memory #1 of DCM#1 (step S805). Then, NVMe SSD$0 transfers L blocks of read data read from Starting LBA based on data address 1 of the read command to memory #0 of assigned CM#0 (step S806).

Next, NVMe SSD$0 reads one block of data following the L blocks of read data based on data address 2 of the read command, and transfers the one block of data that has been read to the doorbell register of FPGA#0 of assigned CM#0 (step S807).

The FPGA#0 of the responsible CM#0 issues an interrupt specifying the MSI-X ID associated with the Doorbell ID of the Doorbell register where writing has been completed (step S808). The CPU#0 of the responsible CM#0 sets the interrupt flag corresponding to the MSI-X ID in the ID correspondence table 400 to "1" (step S809).

Here, let us assume that writing doorbell ID "1" to the doorbell register is complete, and an interrupt specifying MSI-X ID "1" is issued. In this case, as shown in FIG. 9B, the interrupt flag of ID correspondence information 400-2 corresponding to MSI-X ID "1" is set to "1." Note that because the read command specifying SQ ID "0" was completed successfully, ID correspondence information 400-1 has been deleted. Also, ID correspondence information 400-3 corresponding to SQ ID "2" assigned to the newly issued read command has been registered.

When the transfer of the data requested by the read command is completed, NVMe SSD$0 writes a transfer completion notification to the CQ corresponding to CQ ID "1" in memory #1 of DCM#1 (step S810). Then, NVMe SSD$0 issues an MSI-X interrupt to CPU#1 of DCM#1 (step S811).

When CPU #1 of DCM #1 receives the MSI-X interrupt, it confirms the CQ corresponding to CQ ID "1" and completes the process (step S812). Next, CPU #1 of DCM #1 writes the Head value of the CQ to the Completion Queue Doorbell register of NVMe SSD $0 (step S813). The Head value is used to indicate to NVMe SSD $0 that the process is complete.

Then, CPU #1 of DCM #1 sends a completion response (Good) specifying SQ-ID "1" to CPU #0 of responsible CM #0 (step S814). When CPU #0 of responsible CM #0 receives the completion response specifying SQ-ID "1", it checks the interrupt flag corresponding to SQ-ID "1" in ID correspondence table 400 and determines the delivery status of the read data (step S815).

In this case, because the interrupt flag corresponding to SQ-ID "1" is "1", it is determined that the read data has been delivered and the read command specifying SQ ID "1" has been completed successfully. CPU #0 of responsible CM #0 then deletes the entry corresponding to the completed read command from the ID correspondence table 400 (step S816).

Here, as shown in (C) of Figure 9, the ID correspondence information 400-2 (entry) corresponding to SQ-ID "1" is deleted from the ID correspondence table 400.

This makes it possible to ensure delivery of read data even when NVMe SSD$0 is accessed from responsible CM#0 via DCM#1.

Next, the processing procedure when an abnormality occurs in the storage system 100 will be explained using FIG. 10.

Figure 10 is a sequence diagram showing an example of a processing procedure when an abnormality occurs in the storage system 100. Figure 11 is an explanatory diagram (part 2) showing an example of updating the ID correspondence table 400. In the sequence diagram of Figure 10, first, CPU #0 of responsible CM #0 creates ID correspondence information in the ID correspondence table 400 (step S1001).

In this case, the SQ ID to be assigned to the newly issued command is "1". In this case, as shown in FIG. 11(D), ID correspondence information 400-2 corresponding to SQ ID "1" is created and newly registered in ID correspondence table 400. Note that ID correspondence information 400-1 corresponding to SQ ID "0" assigned to the read command being requested for execution is registered in ID correspondence table 400.

Next, CPU#0 of responsible CM#0 specifies the identified SQ ID "1" and sends a request to DCM#j to execute a read command requesting NVMe SSD$k to transfer read data to memory#i (step S1002).

When CPU #1 of DCM #1 receives a request to execute a read command specifying SQ ID "1", it writes the read command to which a request to transfer specified data to the Doorbell register corresponding to SQ ID "1" in FPGA #0 of assigned CM #0 is added, to the SQ corresponding to SQ ID "1" in memory #1 (step S1003).

Next, CPU #1 of DCM #1 writes the Tail value of the SQ to the Submission Queue Doorbell register of NVMe SSD $0 (step S1004). Next, NVMe SSD $0 obtains a read command from the SQ on memory #1 of DCM #1 (step S1005).

Then, NVMe SSD$0 transfers the L blocks of read data read from the Starting LBA to memory #0 of assigned CM#0 based on data address 1 of the read command (step S1006). Next, NVMe SSD$0 reads one block of data following the L blocks of read data based on data address 2 of the read command, and transfers the one block of data that has been read to the doorbell register of FPGA#0 of assigned CM#0 (step S1007).

Here, we will assume that a temporary hardware fault or the like causes the link between PCIe SW #0 of assigned CM #0 and PCIe SW #1 of DCM #1 to be severed, causing data transfer to memory #0 and data transfer to the Doorbell register to fail. In this case, writing to the Doorbell register for Doorbell ID "1" is not executed, so no MSI-X interrupt is issued. For this reason, as shown in FIG. 11 (E), the interrupt flag in ID correspondence information 400-2 corresponding to SQ ID "1" remains at "0."

When the transfer of the data requested by the read command is completed, NVMe SSD$0 writes a transfer completion notification to the CQ corresponding to CQ ID "1" in memory #1 of DCM#1 (step S1008). Then, NVMe SSD$0 issues an MSI-X interrupt to CPU#1 of DCM#1 (step S1009).

When CPU #1 of DCM #1 receives the MSI-X interrupt, it checks the CQ corresponding to CQ ID "1" and completes the process (step S1010). Next, CPU #1 of DCM #1 writes the Head value of the CQ to the Completion Queue Doorbell register of NVMe SSD $0 (step S1011).

Then, CPU #1 of DCM #1 sends a completion response (Good) specifying SQ-ID "1" to CPU #0 of responsible CM #0 (step S1012). When CPU #0 of responsible CM #0 receives the completion response specifying SQ-ID "1", it checks the interrupt flag corresponding to SQ-ID "1" in ID correspondence table 400 and determines the delivery status of the read data (step S1013).

In this case, as shown in FIG. 11(F), the interrupt flag corresponding to SQ-ID "1" is "0", so it is determined that the read data has not been delivered and the read command specifying SQ ID "1" has not been completed successfully. Then, CPU#0 of responsible CM#0 performs a specified recovery process and then resends a request to execute the read command specifying SQ ID "1" to DCM#j (step S1014).

This makes it possible to detect when a data transfer has failed due to a temporary hardware fault, etc. Also, recovery procedures such as reconstructing the path (link) can be performed, and the read command can be re-requested from DCM#j.

Other Embodiments of Storage System 100
Next, another embodiment of the storage system 100 will be described with reference to Fig. 12 and Fig. 13. Here, a case where the storage system 100 includes a plurality of CEs (12 CM configuration) will be described.

Figure 12 is an explanatory diagram (part 1) showing another embodiment of the storage system 100. In Figure 12, the storage system 100 includes CEs $0 to $5. Each of CEs $0 to $5 has two CMs and an NVMe SSD. Note that Figure 12 shows an excerpt of a portion of the hardware configuration of each CM #i.

In the storage system 100, the CMs are connected to each other so that they can communicate with each other via FRT (Front End Router) 1200. The FRT 1200 has a PCIe SW and connects the CMs to each other so that they can communicate with each other. Note that although only FRT 1200 is shown as the FRT connecting the CMs, redundancy may be achieved by using two or more FRTs.

Here, CM#0 is the responsible CM and CM#1 is the DCM. Also, assume that a failure occurs between PCIe SW#0 of responsible CM#0 and NVMe SSD$0, and NVMe SSD$0 cannot be directly accessed from responsible CM#0. In this case, the request to execute a read command from responsible CM#0 to DCM#1 is made via FRT1200.

In addition, data transfer from NVMe SSD $0 to responsible CM #0 is also performed via FRT 1200. Note that in FIG. 12, the thick arrows indicate the path that the read data requested by the host device 110 takes from NVMe SSD #0 to the host device 110.

Next, using FIG. 13, we will explain the case where the responsible CM and the NVMe SSD having the read data are located in different CEs. However, the system configuration of the storage system 100 is the same as the system configuration shown in FIG. 12. Also, FIG. 13 shows an excerpt of a part of the hardware configuration of each CM#i.

Figure 13 is an explanatory diagram (part 2) showing another embodiment of the storage system 100. Here, CM#11 is the responsible CM, and CM#1 (or CM#0) is the DCM. In other words, the NVMe SSD having the read data is NVMe SSD$0 in CE$0.

In this case, the request to execute the read command from the responsible CM #11 to the responsible CM #1 is made via FRT 1200. Data transfer from NVMe SSD $0 to the responsible CM #11 is also made via FRT 1200. Note that in FIG. 13, the thick arrows indicate the path that the read data requested by the host device 110 takes from being read from NVMe SSD #0 to reaching the host device 110.

As described above, according to the embodiment, when CM#i (responsible CM) accesses NVMe SSD$k via DCM#j, it can request NVMe SSD$k to transfer read data to its own memory#i, and can also send a request to DCM#j to execute a read command that requests the transfer of specified data to the doorbell register corresponding to the specified ID (SQ ID) in its own FPGA#i.

As a result, if the NVMe SSD$k cannot be accessed directly, it is possible to request the transfer of read data to the local memory #i and to request the execution of a read command that requests the transfer of specified data to the Doorbell register corresponding to the specified SQ ID in the local FPGA #i.

In addition, according to CM#i (responsible CM), when an interrupt specifying an ID (MSI-X ID related to the SQ ID) is received from FPGA#i, the interrupt flag corresponding to that ID can be enabled. FPGA#i is an integrated circuit provided in CM#i, and includes a doorbell register for data transfer. When data is written to the doorbell register, an interrupt specifying an ID (MSI-X ID) corresponding to that doorbell register is issued to CPU#i.

This allows you to determine the write status to the Doorbell register in FPGA #i.

In addition, according to CM#i (responsible CM), when the execution of the read command issued from DCM#j to NVMe SSD$k in response to the execution request, which requests the transfer of read data to its own memory #i and also requests the transfer of specified data to the doorbell register corresponding to the specified ID (SQ ID), can be completed, the CM#i can receive a completion response specifying the ID (SQ ID) from DCM#j.

This makes it possible to know that the execution of the read command requested to DCM#j has been completed. However, because this is a PCIe posted write, the completion response from DCM#j alone does not indicate whether the transfer of the read data to memory#i was successful.

In addition, when CM#i (responsible CM) receives a completion response from DCM#j specifying an ID (SQ ID), it can refer to the interrupt flag corresponding to that ID to determine the status of delivery of the read data to its own memory#i.

This makes it possible to determine the status of read data delivery to memory #i from the write status (data delivery status) to the Doorbell register in FPGA #i, which is performed using the same path as the data transfer from NVMe SSD $k to memory #i.

CM#i (responsible CM) also includes a doorbell register corresponding to each of the multiple IDs (0 to N) (doorbell ID), and is equipped with an FPGA#i that issues an interrupt specifying the ID corresponding to the doorbell register when a write is made to the doorbell register, and can send a request to execute a read command to DCM#j by specifying an ID (SQ ID) among the multiple IDs that does not correspond to other read commands.

As a result, in the sequence of events relating to each read command, the same ID that does not correspond to other read commands can be used for the SQ, CQ, Doorbell register, and MSI-X interrupts, allowing association by the SQ, CQ, Doorbell register, and MSI-X IDs. Therefore, even when multiple read commands are executed, CPU#i can determine which SQ ID read command completed successfully by checking the MSI-X ID received from FPGA#i.

In addition, according to CM#i (the CM in charge), when a completion response specifying an ID (SQ ID) is received from DCM#j, if the interrupt flag corresponding to that ID is enabled, it can be determined that the read data has been sent to its own memory #i. In addition, according to CM#i (the CM in charge), if the interrupt flag corresponding to that ID is disabled, it can be determined that the read data has not been sent to its own memory #i.

As a result, if a temporary hardware failure occurs and the doorbell register in FPGA #i is not written to, it can be determined that data transfer to its own memory #i was also not successful. On the other hand, if the doorbell register in FPGA #i is written to, it can be determined that data transfer to its own memory #i was also successful.

In addition, according to CM #i (DCM), when another CM #j (responsible CM) accesses NVMe SSD $k via its own CM, it can accept a request from the other CM #j to transfer read data to the memory #j of the other CM #j from the NVMe SSD $k, and a request to execute a read command that requests the transfer of specified data to a doorbell register corresponding to a specified ID in the FPGA #j of the other CM #j. Then, according to CM #i (DCM), it can issue a read command to NVMe SSD $k in response to the execution request from the other CM #j. In addition, according to CM #i (DCM), when there is a notification of completion of the transfer of data requested by the read command from NVMe SSD $k, it can send a completion response specifying the ID to the other CM #j.

In this way, in response to an execution request from the responsible CM #j, a read command requesting the transfer of read data to another memory #j can be issued to the NVMe SSD $k by adding a request to transfer specific data to the doorbell register of the doorbell ID associated with the SQ ID corresponding to the read command. In addition, when the transfer of the data requested by the read command by the NVMe SSD $k is completed, a completion response indicating that the execution of the read command is completed can be notified to the other CM #j.

Specifically, for example, DCM#i sets the destination address of the data area in memory #j of responsible CM#j to data address 1 of the read command. DCM#i also sets the address of the Doorbell register corresponding to the specified SQ-ID in FPGA#j of responsible CM#j to data address 2. DCM#i also sets the Number of Logical Blocks as "L+1". This makes it possible to request the transfer of one block of data to the Doorbell register after the transfer of read data to memory #j is complete, making it possible to effectively determine the delivery status of the read data while suppressing an increase in communication volume.

According to CM#i (DCM), when another CM#j (responsible CM) accesses NVMe SSD$k via its own CM, it is possible to create in its own memory #i an SQ in which a command corresponding to that ID is stored and a CQ in which information indicating the completion status of that command is stored for each of a plurality of IDs (0 to N). Then, according to CM#i (DCM), when an execution request is received from another CM#j, it is possible to store a read command in an SQ corresponding to a specified ID (SQ ID) in its own memory #i. Also, according to CM#i (DCM), when information indicating the completion of a read command is written by NVMe SSD$k to a CQ corresponding to that ID in its own memory #i, it is possible to send a completion response specifying that ID to the other CM#j.

This makes it possible to distinguish the completion status of each read command even when multiple read commands are requested to be executed.

As a result, according to the storage system 100 and CM#i of the embodiment, even when the responsible CM#i accesses the NVMe SSD$k via another CM#j (DCM), it is possible to ensure data delivery guarantee, and data corruption in the host device 110 due to data loss (non-delivery) can be prevented.

The delivery status determination method described in this embodiment can be realized by executing a prepared program on a computer such as a storage control device. This delivery status determination program is recorded on a computer-readable recording medium such as a hard disk, a flexible disk, a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disk), or a USB (Universal Serial Bus) memory, and is executed by being read from the recording medium by the computer. In addition, this delivery status determination program may be distributed via a network such as the Internet.

The following additional notes are provided with respect to the above-described embodiment.

(Note 1) A reconfigurable integrated circuit or a dedicated integrated circuit includes a buffer area for data transfer, and issues an interrupt specifying an identifier corresponding to the buffer area when data is written to the buffer area;
When accessing a storage device that performs data transfer without requesting a response via another storage control device, a request is sent to the other storage control device to request transfer of read data to its own memory, and a request is sent to the other storage control device to execute a read command that requests transfer of predetermined data to a buffer area corresponding to a specified first identifier in the integrated circuit;
when an interrupt specifying the first identifier is received from the integrated circuit, an interrupt flag corresponding to the first identifier is enabled;
receiving, when execution of the read command issued from the other storage control device to the storage device in response to the execution request is completed, a completion response specifying the first identifier from the other storage control device;
when the completion response is received, referring to an interrupt flag corresponding to the first identifier, to determine a delivery status of the read data;
A storage control device comprising a control unit.

(Supplementary Note 2) The integrated circuit comprises:
a buffer area corresponding to each of a plurality of identifiers, and when writing is performed to the buffer area, an interrupt is issued specifying the identifier corresponding to the buffer area;
The control unit is
specifying a first identifier, among the plurality of identifiers, that does not correspond to another read command, and transmitting an execution request for the read command to the other storage control device;
2. The storage control device according to claim 1 .

(Additional Note 3) The control unit
when a completion response designating the first identifier is received from the other storage control device, if an interrupt flag corresponding to the first identifier is enabled, it is determined that the read data has been delivered, and if the interrupt flag corresponding to the first identifier is disabled, it is determined that the read data has not been delivered.
3. The storage control device according to claim 1 or 2.

(Additional Note 4) The control unit
When another storage control device accesses the storage device via the storage control device, the storage control device requests the transfer of read data to a memory of the other storage control device and also receives a request to execute a read command requesting the transfer of predetermined data to a buffer area corresponding to a specified second identifier in an integrated circuit of the other storage control device;
When the execution request is received, the read command is issued to the storage device;
when a transfer completion notification of the data requested by the read command is received from the storage device, a completion response designating the second identifier is transmitted to the other storage control device;
4. The storage control device according to claim 1,

(Additional Note 5) The control unit
When another storage control device accesses the storage device via the storage control device, a first queue is created in the storage control device's memory for each of a plurality of identifiers, in which a command corresponding to the identifier is stored, and a second queue is created in which information indicating a completion state of the command is stored;
When the execution request is received, the read command is stored in a first queue corresponding to the second identifier in the local memory;
when information indicating completion of the read command is written by the storage device to a second queue in its own memory corresponding to the second identifier, transmitting a completion response specifying the second identifier to the other storage control device;
5. The storage control device according to claim 4.

(Appendix 6) The storage control device described in any one of appendices 1 to 5, characterized in that the integrated circuit is an FPGA (Field Programmable Gate Array).

(Appendix 7) The storage control device according to any one of appendices 1 to 6, characterized in that the storage device is an NVMe (Non-Volatile Memory Express) SSD (Solid State Drive).

(Appendix 8) The storage control device described in appendix 1, characterized in that the specified data is transferred to a buffer area corresponding to the first identifier after the transfer of the read data from the storage device to its own memory is completed.

(Supplementary Note 9) A storage control device including a buffer area for data transfer, which issues an interrupt specifying an identifier corresponding to the buffer area when writing to the buffer area, the storage control device including an integrated circuit whose internal circuitry can be reconfigured or a dedicated integrated circuit,
When accessing a storage device that performs data transfer without requesting a response via another storage control device, a request is sent to the other storage control device to request transfer of read data to its own memory, and a request is sent to the other storage control device to execute a read command that requests transfer of predetermined data to a buffer area corresponding to a specified first identifier in the integrated circuit;
when an interrupt specifying the first identifier is received from the integrated circuit, an interrupt flag corresponding to the first identifier is enabled;
receiving, when execution of the read command issued from the other storage control device to the storage device in response to the execution request is completed, a completion response specifying the first identifier from the other storage control device;
when the completion response is received, referring to an interrupt flag corresponding to the first identifier, to determine a delivery status of the read data;
A delivery status determination program that executes a process.

(Supplementary Note 10) A storage system including a plurality of storage control devices and a storage device that performs data transfer without requiring a response,
Any one of the storage control devices,
an integrated circuit having an internal circuit that can be reconfigured or a dedicated integrated circuit that includes a buffer area for data transfer, and issues an interrupt specifying an identifier corresponding to the buffer area when writing is performed to the buffer area;
When accessing a storage device that performs data transfer without requesting a response via another storage control device, a request is sent to the other storage control device to request transfer of read data to its own memory, and a request is sent to the other storage control device to execute a read command that requests transfer of predetermined data to a buffer area corresponding to a specified first identifier in the integrated circuit;
The other storage control device includes:
When the execution request is received, the read command is issued to the storage device;
The storage control device includes:
when an interrupt specifying the first identifier is received from the integrated circuit, an interrupt flag corresponding to the first identifier is enabled;
when execution of the read command issued from the other storage control device to the storage device in response to the execution request is completed, a completion response specifying the first identifier is received from the other storage control device;
when the completion response is received, referring to an interrupt flag corresponding to the first identifier, to determine a delivery status of the read data;
A storage system comprising:

(Additional Note 11) The other storage control device,
creating, for each of the plurality of identifiers, a first queue in which a command corresponding to the identifier is stored and a second queue in which information indicating a completion state of the command is stored in the local memory;
When the execution request is received, the read command is stored in a first queue corresponding to the first identifier in the local memory;
when information indicating completion of the read command is written by the storage device to a second queue in its own memory corresponding to the first identifier, transmitting a completion response specifying the first identifier to the storage control device;
11. The storage system according to claim 10,

100 Storage system 110 Host device 400 ID correspondence table 501 First reception unit 502 First request processing unit 503 Judgment unit 504 Recovery unit 601 Second reception unit 602 Second request processing unit 1200 FRT
cmd NVMe command #0, #1, #i, #j CM, CPU, memory, CA, FPGA, PCIe SW
$0~$5 CE, NVMe SSD

Claims (9)

an integrated circuit having an internal circuit that can be reconfigured or a dedicated integrated circuit that includes a buffer area for data transfer, and issues an interrupt specifying an identifier corresponding to the buffer area when writing is performed to the buffer area;
when accessing a storage device that performs data transfer via another storage control device and does not require a response from a transfer destination for the data transfer, sending to the other storage control device a request to execute a read command that requests the storage device to transfer read data to its own memory and requests the storage device to transfer predetermined data to a buffer area corresponding to a specified first identifier in the integrated circuit;
when an interrupt specifying the first identifier is received from the integrated circuit, an interrupt flag corresponding to the first identifier is enabled;
receiving, when execution of the read command issued from the other storage control device to the storage device in response to the execution request is completed, a completion response specifying the first identifier from the other storage control device;
when the completion response is received, referring to an interrupt flag corresponding to the first identifier, to determine a delivery status of the read data;
A storage control device comprising a control unit. The integrated circuit comprises:
a buffer area corresponding to each of a plurality of identifiers, and when writing is performed to the buffer area, an interrupt is issued specifying the identifier corresponding to the buffer area;
The control unit is
specifying a first identifier, among the plurality of identifiers, that does not correspond to another read command, and transmitting an execution request for the read command to the other storage control device;
The storage control device according to claim 1 . The control unit is
when a completion response designating the first identifier is received from the other storage control device, if an interrupt flag corresponding to the first identifier is enabled, it is determined that the read data has been delivered, and if the interrupt flag corresponding to the first identifier is disabled, it is determined that the read data has not been delivered.
3. The storage control device according to claim 1 or 2. The control unit is
When another storage control device accesses the storage device via the storage control device, the storage control device requests the storage device to transfer read data to a memory of the other storage control device, and also receives a request to execute a read command from the other storage control device to request the storage device to transfer specified data to a buffer area corresponding to a specified second identifier in an integrated circuit of the other storage control device;
When the execution request is received, the read command is issued to the storage device;
when a transfer completion notification of the data requested by the read command is received from the storage device, a completion response specifying the second identifier is transmitted to the other storage control device;
4. The storage control device according to claim 1, wherein the storage control device is a storage control device. The control unit is
When another storage control device accesses the storage device via the storage control device, a first queue is created in the storage control device's memory for each of a plurality of identifiers, in which a command corresponding to the identifier is stored, and a second queue is created in which information indicating a completion state of the command is stored,
When the execution request is received, the read command is stored in a first queue corresponding to the second identifier in the local memory;
when information indicating completion of the read command is written by the storage device to a second queue in its own memory corresponding to the second identifier, transmitting a completion response specifying the second identifier to the other storage control device;
The storage control device according to claim 4 . The storage control device according to any one of claims 1 to 5, characterized in that the integrated circuit is an FPGA (Field Programmable Gate Array). The storage control device according to any one of claims 1 to 6, characterized in that the storage device is a NVMe (Non-Volatile Memory Express) SSD (Solid State Drive). A storage control device including a buffer area for data transfer, which issues an interrupt specifying an identifier corresponding to the buffer area when writing is performed to the buffer area, the storage control device including an integrated circuit whose internal circuitry can be reconfigured or a dedicated integrated circuit,
when accessing a storage device that performs data transfer via another storage control device and does not require a response from a transfer destination for the data transfer, sending to the other storage control device a request to transfer read data to the storage device's own memory and a read command request to transfer predetermined data to a buffer area corresponding to a specified first identifier in the integrated circuit;
when an interrupt specifying the first identifier is received from the integrated circuit, an interrupt flag corresponding to the first identifier is enabled;
receiving, when execution of the read command issued from the other storage control device to the storage device in response to the execution request is completed, a completion response specifying the first identifier from the other storage control device;
when the completion response is received, referring to an interrupt flag corresponding to the first identifier, to determine a delivery status of the read data;
A delivery status determination program that executes a process. A storage system including a plurality of storage control devices and a storage device that performs data transfer and does not require a response from a transfer destination to the data transfer ,
Any one of the storage control devices,
an integrated circuit having an internal circuit that can be reconfigured or a dedicated integrated circuit that includes a buffer area for data transfer, and issues an interrupt specifying an identifier corresponding to the buffer area when writing is performed to the buffer area;
When accessing the storage device via another storage control device, a request is sent to the other storage control device to request the storage device to transfer read data to its own memory, and a request is sent to the other storage control device to execute a read command to request the storage device to transfer predetermined data to a buffer area corresponding to a specified first identifier in the integrated circuit;
The other storage control device includes:
When the execution request is received, the read command is issued to the storage device;
The storage control device includes:
when an interrupt specifying the first identifier is received from the integrated circuit, an interrupt flag corresponding to the first identifier is enabled;
when execution of the read command issued from the other storage control device to the storage device in response to the execution request is completed, a completion response specifying the first identifier is received from the other storage control device;
when the completion response is received, referring to an interrupt flag corresponding to the first identifier, to determine a delivery status of the read data;
A storage system comprising:

Images