HK1182500A - Systems and methods for an enhanced controller architecture in data storage systems - Google Patents

Description

System and method for enhanced controller architecture in data storage systems

Technical Field

The present disclosure relates to non-volatile storage systems including, but not limited to, flash drives. More particularly, the present disclosure relates to systems and methods for enhanced controller architecture in solid state drives.

Background

There are various controller architectures for controlling flash media. The Open NAND Flash Interface (ONFI) is a standard interface that specifies some common command set that flash manufacturers should support. ONFI supports certain low-level, basic I/O operations that can include, for example, page writes/reads and block erases. However, efficient flash media management often involves several functions that are high-level and potentially processing intensive, such as logical-to-physical mapping, garbage collection, and wear leveling. These functions are beyond the ONFI range, and therefore an efficient controller architecture needs to handle these requirements while providing a high level of data throughput performance to the host.

Disclosure of Invention

Drawings

Systems and methods embodying various features of the invention will now be described in detail with reference to the following drawings, in which:

fig. 1A-1C illustrate several Solid State Drive (SSD) controller architectures.

FIG. 1D is a block diagram illustrating a controller architecture according to one embodiment.

Fig. 2A and 2B are block diagrams illustrating controller architectures according to some embodiments.

FIG. 3 is a block diagram illustrating command processing components between the controller and the bridge device, according to one embodiment.

FIG. 4 is a flow diagram illustrating a data processing sequence according to one embodiment.

FIG. 5 illustrates example data ports and how each port corresponds to a different command tag.

FIG. 6 shows example out-of-order commands and the flash memory die they are accessing.

Detailed Description

While certain embodiments of the present invention have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the invention. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms. Also, various omissions, substitutions and changes in the methods and systems described herein may be made without departing from the spirit of the inventions.

I. Controller/bridge design

Fig. 1A-1C illustrate several SSD controller architectures. Fig. 1A shows an SSD controller including an ONFI (open NAND flash interface) interface. The ONFI interface is a low-level parallel I/O interface that provides basic command support to enable external components, such as a host controller, to control operations in the NAND. Fig. 1A illustrates a typical setup in which a host device, such as a computer, includes an SSD controller, which in turn includes an ONFI interface for controlling an ONFI-enabled non-volatile memory (NVM) unit. The ONFI interface provides some basic level of control over operations such as page programming, page reading, block erasing, etc. The current version of ONFI supports one command per die and plane and provides some basic queue capabilities through commands such as "read multiple" and "write multiple" commands. However, this may not have a mix of different command types. In addition, commands are batch processed, which means that the queue must be completely cleared before more commands are accepted. In this manner, the SSD controller may perform high-level NVM management functions such as garbage collection, wear leveling, mapping, etc.

FIG. 1B shows an alternative approach in which the NVM is coupled to a bridge controller/device that performs basic channel management and signal processing at some level of NAND. The bridge may provide the SSD controller with ONFI or an ONFI equivalent interface. However, this interface may be modified from the standard ONFI interface and may support additional functionality, for example, to support multiple commands. In such a design, the SSD controller may still perform high-level NAND management functions, such as garbage collection, and communicate with the bridge over the ONFI or ONFI-equivalent interface.

FIG. 1C illustrates a third approach for coupling an NVM to a bridge in a memory system. In this approach, the bridge performs high-level NVM management functions such as garbage collection, wear leveling, and mapping, and low-level functions such as channel management and signal processing. The storage system is coupled with a host device through a high-level I/O interface, such as eMMC or UFS. This is a common design seen in many memory card products. The host sends commands (e.g., read and write commands) to the storage system using logical addressing. Features such as advanced command queuing, statement of health, and detailed error reporting may be supported.

II System overview

FIG. 1D is a block diagram illustrating a controller design architecture according to some embodiments of the invention. Fig. 1D illustrates an SSD controller performing high-level NVM management functions such as garbage collection, wear leveling, etc. In one embodiment, the SSD controller is coupled with the NVM storage system through a high-level, high-speed interface, such as PCIe and eMMC. Instead of PCIe or eMMC, other standard and/or proprietary interfaces may be extended for use as this bus interface. The NVM storage system in one embodiment includes a bridge that communicates with the SSD controller over a high-level, high-speed interface and controls the NAND memory with a low-level interface such as ONFI. As shown, additional features such as advanced command queuing, statement of health, and detailed error reporting may be supported through the high-level interface.

Unlike the designs described above, the controller in such an architecture provides a rich set of physical control levels (e.g., page level control) for individual elements of the NVM through a complex fast interface such as PCIe. It should be noted that in many controller-bridge type designs, the bridge is typically implemented on the processor with reduced performance due to power issues, while the controller is typically in an environment facing less power concerns. If processor intensive functions migrate to a higher execution capacity controller, the overall latency can be reduced. Thus, in some embodiments, the controller is typically implemented in a higher power processor capable of supporting advanced NVM management functions. On the other hand, in some embodiments, the bridge is implemented in a lower power processor to minimize the energy consumption of the overall NVM memory module/cell. As a result, the bridge can perform basic signal processing and channel management of the NVM, as well as some basic error correction functions and XOR (exclusive OR) parity accumulation. In some embodiments, the controller performs logical to physical address mapping, garbage collection, wear leveling, parity management (through control of the parity accumulator in the bridge), RAID striping, and the like. This division of labor still provides the controller with direct, physical control (e.g., page level) of the NVM, resulting in the controller managing the NVM at both page and block levels through a fast, high-level interface such as PCIe. The controller in one embodiment also manages other synthesis services such as XOR parity accumulators in the bridge.

In one embodiment, another advantage of management task partitioning for this architecture relates to trends in the NVM industry. As the most common types of NVM, such as MLC (multi-level memory cell) NAND, continue to evolve to provide higher capacity at the expense of reduced endurance, physical management of NVM becomes increasingly important. For example, MLC products with 5000P/E (program/erase) -times endurance are being replaced today by next generation MLC products with 1500-2000P/E-times endurance. In general, bridge designers are best able to understand the physical characteristics of the NVM and how to optimally extend its lifetime by implementing various endurance enhancement/management functions. Because of this rapidly changing technology landscape, and because each individual NVM manufacturer may require different such endurance enhancement/management functions, these functions may continue to require fine-tuning to accommodate different and continuing changes in NVM products. Thus, the architecture of certain embodiments provides another advantage of its division by isolating these functions in the bridge and allowing the controller designer to focus on the high-level data management functions. In other words, since the controllers and bridges have different design constraints and priorities, each within the architecture can be upgraded according to different schemes and manners without requiring a complete redesign of the whole.

With the reduced delay provided by this design, the bridge can be paired with a cheaper medium. For example, the bridge may be paired with MLC NAND rather than SLC (Single level cell) NAND while still meeting customer-requested performance metrics. Additionally, in some embodiments, the controller-bridge design described above can be adapted for use in a hybrid drive that includes a flash memory and a hard disk assembly. In those embodiments, the controller manages data access to one or more hard disk drives in addition to data access to the NVM through the bridge. Additional features of this design will be further described below using various figures and descriptions of embodiments of the invention.

II.A controller-bridge implementation

FIG. 2A illustrates an embodiment of the controller-bridge architecture previously introduced in FIG. 1D. As shown, solid state non-volatile memory system 120 is connected to host system 110. The host system 110 communicates with the non-volatile storage system 120 using a storage interface 112. The host storage interface 112 is capable of communicating with the nonvolatile storage system 120 using any known communication protocol, such as SATA, SCSI, SAS, USB, fibre channel, PCIe, eMMC, and the like.

In one embodiment, non-volatile storage system 120 includes controller 130 and NVM storage module 150. The controller 130 in one embodiment communicates (through the bus logic/interface 140) with a bridge device 152 within the NVM storage module 150 via a high-level interface such as PCIe. PCIe is used in one embodiment because it defines a rich packet-based routing and quality of service (QOS) infrastructure and provides a high speed interface. The controller may include a processor 136 that controls data functions, and the cores may be coupled to static memory 132 and dynamic memory 134. The processor 130 may also include a data path 138 for processing/transferring data associated with data access commands from the host system 110. In one embodiment, controller 130 is implemented on a SoC (system on a chip), although those skilled in the art will recognize that other hardware/firmware implementations are possible.

In one embodiment, the use of PCIe means that address ranges assigned to device functions are used to construct packet routes both on and within the device. In one embodiment, the PCIe transaction layer delivers the packet to an internal register interface that is read by the firmware. Advanced devices often direct incoming packets to internal RAM or hardware acceleration modules.

The bridge device 152 in one embodiment includes bus logic/interface 154 for communicating with the bus logic/interface 140 (on the controller 130) over a high-level interface bus. At the other end of the bridge, bridge device 152 includes a low level interface 158, such as ONFI for communicating with NVM memory 160 (e.g., NAND), which may include several storage devices, such as flash memory dies 162, 164, 166, and 168. Although ONFI is shown in this embodiment, other suitable flash memory interfaces may be used. In another embodiment, the bridge may use a different interface, such as Toggle or a proprietary interface, to communicate with NVM memory 160 or send boot commands to the memory.

II.B division of work

The advantages of partitioning the NVM management functions are outlined in section II. In particular, the architecture reduces latency and handles various design constraints while allowing the controller and bridge designer to optimize their respective portions of the architecture. In one embodiment, the controller is responsible for block level management, parity strip placement, garbage collection, wear leveling, handling read disturb, and error recovery. In one embodiment, the bridge device manages the raw NVM flash interface. It may also provide one or more of the following functions: command queuing, error correction, XOR parity accumulator, data protection, and improving block persistence. In one embodiment, the interface between the bridge and the controller is a lightweight, PCIe-based data and management interface. The controller configures the bridge and data commands using the interface control commands to access the NVM media.

It should also be noted that the controller uses physical page addressing instead of logical page addressing, which is common in existing controller-bridge designs. The bridge may identify the relationship between pages, blocks, planes, and dies. This provides the controller with maximum flexibility to create a RAID stripe layout, perform data moves, and handle bad blocks. These details are extracted from the bridge. When direct addressing is used, the controller simply provides the bridge with a set of direct page addresses in the command header. In one embodiment, pages are not necessarily contiguous, or even in the same block. In most cases, the controller will access pages across multiple planes and multiple dies in order to maximize concurrent hardware access.

II.C hybrid applications

Certain embodiments of the controller-bridge architecture can be adapted for other uses. For example, FIG. 2B illustrates the use of a controller architecture in a hybrid drive 122, where the hybrid drive 122 includes, in addition to the NVM and bridge components described above, a magnetic storage module 180 having magnetic media 184, such as a rotating Hard Disk Drive (HDD). Controller 130 in this embodiment would therefore manage data access to both NVM storage module 150 and magnetic storage module 180. In one embodiment, an interface other than interface 140 (which connects to NVM) may be used to connect controller 130 to magnetic storage module 180.

Hybrid applications show additional advantages of the controller architecture. A hybrid drive typically includes an SSD with its own internal controller with a mapping table to address the NVM within the SSD. While the HDD part of the hybrid (controller) is usually directly addressed, the hybrid controller uses a special mapping table to determine whether the data is in the SSD or HDD. The use of this special mapping table with the internal SSD mapping table incurs a double overhead in case of access to data in the SSD part of the hybrid drive, due to the two mapping tables and the significant cost associated with maintaining each table.

In contrast, because the controller 130 in the existing architecture manages NVM and magnetic media at both the block level and the page level, it is able to provide consistent address management across flash memory and magnetic media in a single location. Thus, it is not necessary to have both tables as described above. This has the advantage of reducing double table lookups and all associated costs/complexities involved in maintaining separate mapping tables. Direct page addressing is used in a unified mapping scheme.

In addition, in hybrid applications, even if the NVM has a large number of bad blocks (e.g., 50%), it can still provide effective performance enhancement. In a hybrid embodiment, the controller also has efficient address gap handling capability (for gaps caused by bad blocks). In alternative hybrid embodiments, a uniform addressing scheme does not necessarily require the bridge to work with the controller. The controller may access the NVM using the original NVM interface (e.g., ONFI).

Data Command processing

FIG. 3 is a block diagram illustrating command processing components between the controller and the bridge device, according to one embodiment. With the construction of a PCIe interface (or its equivalent interface), both the controller and the bridge implement their own address spaces (210, 250) in their respective device memories that can be accessed by other devices. In one embodiment, messages are delivered by writing the messages to queues located within certain addresses within the address space, and the addresses are stored in the configuration and status registers 252. The use of separate queues for processing data access commands and communications between the controller and the bridge is described further below.

III.A Command and management queue-bridge

In one embodiment, the controller sends a data access command to a command queue 262 in the bridge device. This is performed by the controller sending a data command message to the bridge (by writing to the command queue BAR (base address register)). In one embodiment, the command queue has space for 16 messages, although the number of messages may vary in other embodiments. The command queue can be implemented in several ways. One option is full hardware automation in the case where the controller writes only to a fixed offset. Alternatively, it may be implemented in memory using a circular buffer or an array-based linked list. In one embodiment, the implementation must allow for efficient insertion and advertisement with minimal bus traffic. In one embodiment, the controller knows the current queue length based on the number of status responses that the bridge has sent back (e.g., a message to the controller's completion queue indicates command completion). It should be noted that the data commands are much smaller than the actual data. Once the bridge sends back a completion status or error report, a given recording gap in the queue is considered available.

In the embodiment illustrated in FIG. 3, the bridge side 250 also implements a Configuration and Status Register (CSR) 252 and a management queue 258, the management queue 258 for receiving command messages from the controller related to the operation of the command queue (e.g., messages for suspending the command queue) or management messages generally related to the operation of the bridge. The management queue 258 may be implemented in a manner similar to a command queue, such as by full hardware automation or a circular buffer. Also, like the command queue, the management queue may be configured for efficient insertion and notification with minimal bus traffic. Like the command queue, the controller may derive the current queue length and available gaps based on responses from the bridge.

III.B State queue controller

On the controller side 210 are a set of data ports 214 at data addresses 212 and several status queues. In one embodiment, the status queues include error queue 218, information queue 222, and command completion queue 226. These queues are responsible for receiving messages from the bridge regarding command processing and the current state of the bridge and NVM. Additional details regarding the operations on these queues are further described in sections V and VI below.

IIICommunication between C controller and bridge

In one embodiment, communication between the controller and the bridge is implemented through a PCIe protocol stack 230, wherein the PCIe protocol stack 230 includes several layers on two sides, including a transaction layer (232, 242), a data link layer (234, 240), and a physical layer (236, 238). Although PCIe is used in this disclosure to illustrate the operation of the controller and bridge, other similar standards may be used.

The PCIe transaction layer allocates transmission credits based on how much space is left in its Virtual Channel (VC) buffer space. According to the PCIe specification, devices must implement VC0, although some devices implement additional VCs to ensure that high priority messages have dedicated resources. Packets are directed to the appropriate VC based on their Traffic Class (TC). TCs are also used to determine priority when packets flow over the PCIe fabric. Higher TC packets are typically given priority by the root complex, the switch and the end devices.

In one embodiment, the controller is designed to operate only with VC 0. In one embodiment, although the bridge may implement additional VCs, it must be configurable so that it can operate in a single VC mode. Messages passed between the controller and the bridge will be better understood in view of the following brief description of the data processing flow. To service a read command from the host, the controller may first send a command message to the bridge's command queue. Once the bridge processes the command message, it will read the requested data from the NVM and send the read data back to the corresponding data port on the controller side. This action triggers the data path on the controller, which causes the data to be sent back to the host. Instead, to service a write command from the host, the controller may first send a command message to the bridge's command queue. Once the bridge processes the command message, it reads from the corresponding data port on the controller side. This action triggers the data path on the controller, which causes the write data to be sent from the buffer in the controller to the bridge for writing to the NVM.

In one embodiment, the controller communicates with the bridge using three message types of increasing priority: data written to NVM for write command (0), message for bridge command queue (1), and message for bridge management queue (2). Those skilled in the art will recognize that these messages may be assigned different priorities and may be combined into fewer types or divided into more types depending on the implementation. In one embodiment, the controller sends a steady stream of packets to the bridge under normal conditions.

In one embodiment, the bridge interacts with the controller using its own set of prioritized message types (listed here with increasing priority): data read from NVM for read command (0), message for controller completion/information queue (1), and message for controller error queue (2). Those skilled in the art will recognize that these messages may be assigned different priorities, and may be combined into fewer types or divided into more types, depending on the implementation. As will be described further below, to facilitate fast processing of data access commands, reading or writing of data ports in the controller by the bridge automatically triggers the data path in the controller. In one embodiment, it is not uncommon for a bridge to process multiple commands in parallel. In one embodiment, the bridge notifies the controller with completion queue 226 when the command has completed successfully. In addition, when a detailed error report is sent to the error queue 218, non-critical messages are sent to the message queue 222. In other embodiments, these queues may be combined into fewer queues (with different message types distinguished by special flags or implied address values) or split into more queues (e.g., different error queues for different types of errors or different information queues for different types of information returned from the bridge).

In other embodiments using an interface other than PCIe, the PCIe protocol stack may be replaced with the appropriate stack/layer of the interface. Those skilled in the art will recognize that other equivalent standardized interfaces (e.g., eMMC) may be suitable in place of PCIe. In other embodiments, a custom/proprietary interface may be used to handle communication between the controller and the bridge.

Implied command tag ID and trigger data path

FIG. 4 is a flow diagram illustrating a data processing sequence according to one embodiment. As described above, in one embodiment, the controller initiates the sequence in block 302 by sending an access command (e.g., read, write, erase) to the command queue 262. The command sent to the command queue may include fields such as a tag field, a priority field, a list of pages, and bits that control the XOR parity accumulator. In addition, some commands may specify the address location from which the parity bits are read, and the address location of the post-operation to which the parity bits are written. Location information may also be provided for RAID parity striping. At block 304, the bridge optimally orders the commands in the queue based on their priorities and the bridge's current workload. At block 306, when the bridge is ready to begin working on a given command, it performs a read or write operation on the appropriate data port 214 to trigger a data path on the controller side. In particular, in one embodiment, the datapath includes logic for processing data transferred between the controller and the bridge. For example, for a write, the write data is read from a memory buffer in the controller and processed by the data path (e.g., adding additional metadata) before it is sent to the bridge for writing to the NVM. Similarly, for reads, the datapath also processes input data from the bridge (e.g., removes metadata). The use of a data path on the controller simplifies the overall design and minimizes the work that the bridge needs to do for each command. In view of the above, it is necessary to configure/set the data path for the particular command currently being processed so that the data to be transmitted can be appropriately processed as associated with the current command. This setup/configuration may be performed by some automatic operation in the datapath or by firmware on the controller processor 136. In either scenario, in one embodiment, the bridge read/write data port triggers this configuration of the data path on the controller side. In other embodiments, multiple data paths may be used, each handling a subset of the data ports, although in this scheme the data paths would still operate based on the principles described above.

As shown in block 308, for a read command, the bridge obtains data from the NVM and writes it to the corresponding data port 214, and for a write command, the bridge reads data from the corresponding data port 214 and writes it to the NVM. In other embodiments, other less efficient variations are possible. For example, the controller may read and write transactions, and the bridge may simply notify that attention is needed through an interrupt or the like.

In one embodiment, each command in the bridge command queue 262 has a tag. When the bridge is ready to begin working on a command that involves a data transfer, it accesses the data port 214 that matches the command tag. In one embodiment, the data path in the controller has sixteen ports defined by two values in the bridge CSR, namely the base address and the port size. These two values are sufficient to derive the position of all sixteen ports. In other embodiments, different port numbers and/or different address derivation schemes may be used.

FIG. 5 is a block diagram illustrating how each port corresponds to a different command tag, according to one embodiment. FIG. 5 expands the view of data address 212 from FIG. 3 and shows how the data ports are associated with command tags. As shown, each data port 0 through 15 (having its own uniquely assigned address) is associated with a command tag. In one embodiment, the commands are limited to eight pages. In such an arrangement, the ports need to be at least 64K apart, although the spacing may be greater. Thus, performing a read or write on the port that matches the command tag allows the controller to automatically recognize the matching command and initiate a datapath automation operation without additional control overhead.

For example, when a bridge accesses a particular data port address (e.g., numbers 0 through 15) in the controller defined by the associated PCIe address range, the controller will understand that this is for the command associated with the tag. Thus, the bridge does not need to send command tags separately, which reduces overhead as each additional communication between the bridge and the controller adds to the overall delay. To accomplish this, in one embodiment, the controller automatically decodes the address (splits the higher bits) and loads registers to trigger/prepare the host datapath for processing (initiates automation operations). However, those skilled in the art will recognize that implementations other than the implied tag implementation described above may also be used. For example, the bridge may send an explicit command tag message to the controller to indicate the command the bridge is currently operating on and which data port it intends to use. The preparation of the data path will then depend on the explicit command tag message. In other embodiments, the command tag described above need not be used. In general, any command configuration data that enables the bridge and/or controller to track data ports and command dependencies may be used.

Returning to FIG. 4, in block 310, the controller datapath is automatically triggered when a data port is accessed by the bridge. In one embodiment, once the datapath has been triggered, it must complete the command, since there is no mechanism to allow the bridge to work on part of the command. In one embodiment, when the bridge starts a write operation, the controller sends all pages to be written to the bridge in the order specified by the command message. Instead, the controller may also require the bridge to send data for read operations in the order specified by the command message. Although the examples provided herein show one data path handling both reading and writing, in other embodiments, multiple data paths may be used. For example, in a multiple data lane implementation, each data lane may be dedicated to a subset of the data ports, and/or some data lanes may be configured to handle reads, and other data lanes may be configured to handle writes. Finally, in block 312, the bridge executes the command and returns the status message(s) to one or more queues on the controller side. The queues on the controller side (e.g., completion, info, and error queues) are described in section VI below. In an alternative embodiment, instead of using a tag, the bridge may send a block of data to the controller to program the data path. The bridge does not have to know what the data is doing. The program data block together with the command will first be sent by the controller to the bridge. The bridge will then send the block back. The program data may be sent before the data for the command is to be transmitted, or it may be sent to another queue.

In another embodiment, instead of the implied command tag/datapath trigger mechanism described above, the controller-bridge may communicate in a controller-push (controller-push) mode where the controller sends data to the bridge along with the command. Thus, the bridge may require a large volatile memory buffer capacity in order to hold user data from the controller for various commands in the bridge's command queue. This implementation would reduce latency, but would likely increase the cost of the bridge implementation since it would require the addition of large memory buffers into the bridge. This also results in increased power consumption of the bridge.

V. advanced queuing management

In one embodiment, the bridge supports several queues. As shown in FIG. 3, in one embodiment, the bridge has at least one command queue 262 and one control/management queue 258. The management queue 258 supports queue management commands or other general operational commands from the controller. For example, the controller may send a command to the management queue to request the bridge to suspend processing commands in the command queue or to completely empty the command queue.

In one embodiment, the command queue supports complex queuing and out-of-order execution, while the management queue is in-order. The various queues on both the controller side and the bridge side may have a mix of outstanding commands and may be asynchronous. Command mixing in the bridge command queue is particularly noticeable compared to the ONFI specification. ONFI provides some basic queuing capabilities through its "read multiple" and "write multiple" commands. However, there may be no mix of different types of commands. Also, commands are batch processed, which means that the commands in the queue must be completely emptied before more commands can be accepted.

In contrast, the advanced queuing capability of the bridge is able to (1) accept mixed command types, (2) support out-of-order execution, and (3) allow the controller to send additional commands without first emptying the queue. The bridge may also accept special commands from the controller to specify that a command is executed with high priority. The bridge manages several lanes so it has the flexibility to reorder the commands it receives.

V.A Command ordering

In one embodiment, the command queue may be implemented as a single queue that processes commands of various priority types indicated by queuing flags (e.g., "priority," "in-order," "out-of-order," and "background"), or as several separate queues based on the queuing flags. Data commands may be unordered by default and subject to ordering by the bridge in order to take advantage of hardware optimization and media usage. In one embodiment, the "priority" and "in order" flags are used by the controller to indicate an offset from the default condition.

FIG. 6 shows four example out-of-order commands and the flash memory die they are accessing. Although many permutations are possible, this example of a set of read and write commands is used to illustrate operational interleaving. The bridge requires the factors in the datapath constraints described above in section IV in deciding the order in which to process these commands. Bridges can achieve significant performance advantages by developing intelligent sorting algorithms that can easily recognize non-conflicting commands.

If commands A-D are write commands, then the bridge may maximize concurrency by executing commands A, C and D in parallel (A and B cannot be executed in parallel). The bridge may also pull data for command B down from the controller and work on the portion going into die 2 if the bridge has sufficient buffer space. On the other hand, if commands A-D are read commands, then the bridge may maximize concurrency by executing commands A, C and D in parallel. Although it may read the data for command B on die 2, the bridge may be required to send the data to the controller in the order specified by the command header.

V.B background priority

In one embodiment, the only feature in the queuing mode is the implementation of background priority. Background priority lets the bridge decide when to execute a command. In one embodiment, commands with a "background" flag are unordered and given the lowest priority. They may also be exempt from a command advance timer requirement, which is a time value that specifies the time limit for which a command should be executed. In one embodiment, while the order in which commands are executed is left to the bridge to decide, the commands cannot linger in the queue indefinitely. When the bridge selects between out-of-order commands on the pending list, it may prefer commands with a timeout promotion timer. In one embodiment, the timeout value is set by the controller in the bridge control CSR field.

When a bridge has available resources, it can work on these "background" commands. For example, background priority may be used for data retention and health scan tasks. As another example, the controller may send a block erase command with the background flag set down. When the target die is no longer needed by other commands in the queue, the bridge will perform an erase operation. In one embodiment, the "order", "priority" and "background" flags are mutually exclusive. This context command feature is currently not present in the in-band I/O interface (e.g., UFS, eMMC, SAS, or SATA), since the in-band I/O interface does not support background long running tasks.

Completion, info and error queue

As previously shown in FIG. 3, the controller may also have several queues so that the bridge can return information related to data commands (completion, error, etc.). In addition, the bridge may report other status, errors, and non-critical information (i.e., information/health reports) indicating operations involving the bridge and the NVM. In one embodiment, these queues may be processed sequentially and may be implemented in a memory like a circular buffer with a fixed record size. In one embodiment, the controller implements three status queues to simplify command transactions. When the bridge has successfully completed one or more commands, it indicates using completion queue 226. The information queue 222 is used for non-critical information such as health reports. Error queue 218 allows the bridge to send detailed reports when one or more commands fail. Those skilled in the art will recognize that the three queues may be combined into fewer queues or divided into additional queues. Alternatively, instead of these queues, the controller and bridge may use an interrupt-based system whereby the bridge sends an interrupt signal when the bridge wishes to communicate with the controller. The controller may then examine the messages stored on the bridge side.

In one embodiment, the controller sets the size of each queue in the base address and CSR. In one embodiment, the number of queue entries need not be exchanged, as both the controller and the bridge have enough information to derive this value. In one embodiment, if the controller attempts to configure a queue with less than one entry, the bridge needs to generate an error.

In one embodiment, the bridge is configured to be required to write to the next valid slot and track how many entries it has written. The address for each slot is derived from the start address and the maximum message size. In one embodiment, each queue entry is required to start on a valid boundary. Since the act of writing the last Double Word (DW) is often used to trigger hardware automation, the message is filled to the full record size.

The bridge may write multiple entries into one queue in a single operation. One embodiment implements a doorbell mode in which the controller does not take action on new entries until the bridge writes to the associated doorbell register with a count that it has added a record. In the automatic mode, the controller generates its own signaling when one or more entries are added to the queue. The queue mode (auto or doorbell) may be configured in the CSR.

In one embodiment, the controller-side queue is sized to match the maximum number of potential entries. Generally, these values are proportional to the command queue length of the bridge. Assuming each command has a tag, the controller will not reuse the tag until it has received the state and emptied the queue space.

VI.A completion queue

Assuming that not all commands result in a bus transfer, the controller wishes to be notified when the data and management commands have completed successfully. Attempts to embed state into regular datapath messages create alignment problems and other edge conditions. Instead, in one embodiment, the bridge simply writes a completion notification to the completion queue on the controller side. Although other implementations are possible, in one embodiment, it is sufficient to send a 32-bit Double Word (DW) with each bit set to 1 to represent a command that has completed successfully. For example, in the case of 16 gaps in each of the management queue and the command queue, the upper 16 bits of the DW may be mapped to the management tag, and the lower 16 bits may be mapped to the command tag. Although the bridge may send status immediately, system efficiency increases when multiple completion tags are joined together. Assuming that the length of each of these bridge queues is 16 in this example embodiment, then the completion queue must be 32 entries long. However, in other embodiments, different queue lengths for the two queues are possible, and the completion queue mechanism is adjusted accordingly.

VI.B information queue

In one embodiment, the bridge may send normal system/state information messages to the controller by writing to the information queue 222 (i.e., to the information queue address range 220). The statement-of-health of the NVM is sent to this queue, for example, and other messages are possible. In one embodiment, this queue length is 16 entries. In one embodiment, if there are outstanding read commands, the controller may not issue an active or concurrent health scan. The active concurrent health scan of the NVM is done by the bridge under the direction of the controller. The active scan of the NVM is performed without returning data, while the concurrent scan is performed concurrently with normal data access operations.

VI.C error queue

In one embodiment, the bridge sends an error message to the controller error queue 218 by writing to the error queue address range 216. In one embodiment, this queue length is 16 entries.

XOR parity accumulator management

In one embodiment, the bridge includes an XOR parity accumulator managed by the controller, which makes the data path in the controller simpler. The controller manages the XOR parity accumulator by an XOR parity accumulator command. In one embodiment, the controllers issue common control operation instructions/commands (e.g., embedded within read and write commands), such as: (1) clearing before accumulation (operation: read, write), (2) accumulating parity in a buffer (operation: read, write), (3) writing a parity buffer to a page in the NAND (operation: write). In one embodiment, the instruction/command is passed in three bits in the data access command field. To reduce command size, a dedicated command may be used for less common XOR parity accumulator operations, such as: load parity buffers from pages in the NAND, read parity buffers over the bus, load parity buffers over the bus, and reset all parity buffers.

VIII. other features

In one embodiment, the bridge supports several power management commands. Returning to fig. 2, in one embodiment, the bridge is coupled to and manages a power controller 170, and power controller 170 may include a power throttle controller 172. The bridge may operate at several power budgets specified by the controller 130. The bridge may be set to operate in a throttled state, where the bridge minimizes power consumption with a smaller upper power budget limit. Power management may be implemented through monitored control pins or PCIe-based messaging. In another embodiment, the bridge may use an alternate power source.

Depending on the number of available guarantees currently available for command execution, the bridge may itself implement an energy guarantee based throttling strategy. In the alternative, the controller may implement a policy based on energy guarantees, and the bridge is configured to support power commands issued by the controller based on the policy. One example strategy allows the controller and/or bridge to set a maximum number of concurrent operations and/or a time delay between operations such that the average power consumption stays below a threshold. Various types of Energy-Based warranty-Based strategies are further described in co-pending U.S. application No. 13/216177, filed on 23.8.2011, entitled "Non-volatile Storage sub-system with Energy engineering-Based Performance Throttling," the disclosure of which is incorporated herein by reference. In another embodiment, the bridge is configured to report power consumption for various operations and allow the controller to set explicit limits through exposed interfaces.

In another embodiment, unlike a public bridge implementation, the bridge exposes NAND level information that is normally available on the ONFI interface, but is hidden in other bridge-controller architectures, because many controllers in those architectures are not managed at the physical page level. The following are some example values that the controller can access:

● device manufacturer (ONFI bytes 32-43)

● type device (ONFI byte 44-63)

● JEDEC (Joint development electronic engineering conference) manufacturing ID (ONFI byte 64)

● date code (ONFI byte 65-66)

● bytes of data per page (ONFI bytes 80-83)

● spare bytes per page (ONFI bytes 84-85)

● pages per block (ONFI bytes 92-95)

● number of blocks per die (ONFI bytes 96-99)

● number of die per bridge (New)

● bits per cell (ONFI byte 102)

● maximum number of bad blocks per die (ONFI bytes 103-104)

● block P/E-times durability of MLC (ONFI bytes 105-106)

● SLC Block P/E-times durability (New)

● number of planes (ONFI byte 113)

● longest page Programming time (microseconds) (ONFI bytes 133-

● longest Block Erase time (microseconds) (ONFI bytes 135-)

● longest page read time (microseconds) (ONFI bytes 137-138)

● longest multiplanar page read time (microseconds) (ONFI byte 152-

In some embodiments, because the controller is in a unique location that manages the NVM at both the block and page levels, exposure of these values is useful to assist the controller in managing the NVM.

In one embodiment, the bridge also supports at least some of the configuration details listed below. At least some of these relate to the features mentioned above:

● command a promotion timeout-in one embodiment, when a bridge selects between unordered commands on the pending list, it may prefer commands that have exceeded a promotion timer. As described above, in one embodiment, this timeout does not apply to background commands.

● data Port base — in one embodiment, a host has 16 ports, each corresponding to a message tag. The port size defined below, along with the base address, allows the bridge to derive the address of each data port. These ports are accessed when the bridge executes read and write commands.

● data Port size-in one embodiment, the Port size indicates how much data can be written to each port. In one embodiment, the port is guaranteed to match the maximum command size. In one configuration, the lower two bits are zeros.

● completion queue base-in one embodiment, the bridge sends a command completion notification to this queue.

● completion queue size — in one embodiment, this is the size in bytes. In one configuration, the lower two bits are zeros.

● completion queue doorbell — in one embodiment, the bridge writes its just-added record count to the completion queue. The doorbell is disabled when the doorbell is set to zero and the queue is assumed to be in auto mode.

● completion queue maximum record size-in one embodiment, the maximum message size sent to this queue. In one configuration, this is a read-only value set by the bridge.

● information queue base — in one embodiment, the bridge writes a non-critical report to this address.

● information queue size-in one embodiment, this is the size in bytes. In one configuration, the lower two bits are zeros.

● message queue doorbell-in one embodiment, the bridge writes the record count it just added to the message queue. The doorbell is disabled when the doorbell is set to zero and the queue is assumed to be in auto mode.

● information queue maximum record size-in one embodiment, the maximum message size sent to this queue. This is the read-only value set by the bridge.

● error queue base-in one embodiment, the bridge writes an error report to this address.

● error queue size-in one embodiment, this is the size in bytes. In one configuration, the lower two bits are zeros.

● error queue doorbell — in one embodiment, the bridge writes its just-added record count to the error queue. The doorbell is disabled when the doorbell is set to zero and the queue is assumed to be in auto mode.

● maximum record size of error queue — in one embodiment, this is the maximum message size sent to this queue. This is the read-only value set by the bridge.

In one embodiment, the bridge is capable of supporting SLC mode. That is, the controller may specify that certain portions of the MLC NAND operate as SLC. While this is a useful option in ONFI, many public bridge implementations do not support this capability.

IX. alternative embodiments; conclusion

As used in this application, "non-volatile memory" generally refers to solid state memory such as NAND flash memory. However, the systems and methods of the present disclosure are also useful in more conventional hard disk drives and hybrid drives, including both solid state and hard disk drive components. Thus, while certain internal operations are mentioned as being generally associated with solid state drives, such as "wear leveling" and "garbage collection," similar operations of hard disk drives may also utilize certain embodiments of the present disclosure. Solid state memory may include a variety of technologies such as flash integrated circuits, chalcogenide RAM (C-RAM), phase change memory (PC-RAM or PRAM), programmable metallization cell RAM (PMC-RAM or PMCM), Ovonic Unified Memory (OUM), Resistive RAM (RRAM), NAND memory, NOR memory, EEPROM, ferroelectric memory (FeRAM), or other discrete NVM (non-volatile memory) chips. As is well known in the art, a solid state storage device (e.g., a die) may be physically divided into planes, blocks, pages, and sectors. Other forms of memory (battery backed volatile DRAM or SRAM devices, disk drives, etc.) may additionally or alternatively be used.

While certain embodiments of the present invention have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the invention. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions as described above. For example, the various components shown in fig. 1D, 2A, 2B, and 3 may be implemented in software and/or firmware on a processor, an ASIC/FPGA, or dedicated hardware. For example, those skilled in the art will appreciate that in some embodiments, the actual steps taken in the process shown in FIG. 4 may differ from those shown in the figure. According to embodiments, some steps described in the above examples may be removed, other steps added, and the order of the steps may be changed and/or some steps may be completed in parallel. Moreover, the features and attributes of the specific embodiments disclosed above may be combined in different ways to form additional embodiments, all of which fall within the scope of the present disclosure. While the present disclosure provides certain preferred embodiments and applications, other embodiments that are apparent to those of ordinary skill in the art, including embodiments that do not provide all of the features and advantages described herein, are also within the scope of the present disclosure. Accordingly, the scope of the present disclosure is to be limited only by the following claims.

Claims (32)

1. A non-volatile memory storage system, comprising:

an array of one or more solid state storage devices; and

a bridge device coupled with the array, the bridge device comprising:

a first interface for communicating data access instructions to the array of one or more solid state storage devices; and

a second interface for receiving a physical page level data access command from a controller;

wherein the bridge device is configured to implement a first queue for receiving the data access command from the controller.

2. The non-volatile memory storage system of claim 1, wherein the bridge device is further configured to manage one or more data channels in the array, and to manage concurrent processing of the data access commands based on activity in the one or more data channels.

3. The non-volatile memory storage system of claim 1, wherein the bridge device further comprises:

a second queue for receiving management commands, wherein at least one of the management commands is associated with an operation of the first queue.

4. The non-volatile memory storage system of claim 1, wherein the bridge device is configured to receive data access commands with different priority indications from the controller in the first queue, and

wherein the bridge device is further configured to determine an order in which to process received data access commands, thereby maximizing concurrent execution of commands in the array of one or more solid state storage devices, the order determined according to at least one of: (1) the priority indication and (2) a current state of activity in the array.

5. The non-volatile memory storage system of claim 4, wherein one of the priority indications is a background priority indication that allows the bridge device to execute associated commands without a specific time limit.

6. The non-volatile memory storage system of claim 1, wherein the first interface is ONFI or Toggle.

7. The non-volatile memory storage system of claim 1, wherein the second interface is PCIe.

8. The non-volatile memory storage system of claim 1, wherein the bridge device is further configured to:

selecting one of the data access commands from the controller for processing within the first queue; and

accessing a data port in the controller associated with the selected data access command, thereby causing a data path in the controller to initiate a transfer of data for the data access command.

9. The non-volatile memory storage system of claim 1, wherein the bridge device further comprises an XOR parity accumulator and the bridge device is configured to execute an XOR parity accumulator instruction embedded in at least one of the data access commands.

10. The non-volatile memory storage system of claim 1, wherein the bridge device further comprises an error correction module to correct data errors encountered in execution of the data access command.

11. The non-volatile memory storage system of claim 1, wherein the bridge device is configured to provide the controller with low level state information of the array of one or more solid state storage devices.

12. A controller apparatus for controlling data operations in a non-volatile memory module, the controller apparatus comprising:

at least one data path;

an interface for communicating with a bridge device coupled to the non-volatile storage; and

a processor configured to implement an address range in a memory, wherein the memory comprises addresses of a plurality of data ports configured for data operations, the data ports configured to be accessed by the bridge device, the accesses relating to data access commands sent by the controller device to the bridge device;

wherein at least one data path of the controller device is triggered by an access of the bridge device to one of the data ports.

13. The controller device of claim 12, wherein at least some of the data ports are associated with command configuration information that is also associated with commands issued by the controller device to the bridge device such that access by the bridge device to a particular data port can be correlated to commands having matching command configuration information that were previously issued by the controller to the bridge device.

14. The controller device of claim 13, wherein the command configuration information includes one or more command tags.

15. The controller device of claim 12, wherein addresses of at least some of the data ports function as implicit command tags, whereby when the bridge device accesses one of such data ports, the bridge device can initiate data operations on the at least one data path without a message from the bridge device identifying a command associated with the current data port access.

16. The controller device of claim 12, wherein the processor is further configured to implement at least one of the following within the address range:

a queue for receiving error messages from the bridge device;

a queue for receiving information messages from the bridge device; and

a queue for receiving command completion messages from the bridge device.

17. The controller device of claim 12, wherein the processor is further configured to implement within the address range a queue that receives error messages, information messages, and command complete messages from the bridge device.

18. The controller device of claim 12, wherein the processor is further configured to receive an interrupt from the bridge device that a notification message is pending within the bridge device.

19. The controller device of claim 12, wherein the interface is PCIe.

20. The controller device of claim 12, further comprising a plurality of data lanes, each data lane associated with a subset of the data ports.

21. A method for processing data operations in a controller architecture, wherein the controller architecture comprises a controller device and a bridge device coupled with non-volatile storage, the method comprising performing the following operations in the controller device:

sending a data access command to the bridge device over an interface connecting the controller device and the bridge device, the data access command including a physical address instruction associated with the command.

22. The method of claim 21, further comprising completing the following in the controller device:

in response to the bridge device accessing a data port on the controller device associated with the command, initiating configuration of a data path in the controller to prepare for transfer of data related to the command.

23. The method of claim 22, wherein the configuring of the datapath is accomplished by an automated operation associated with the datapath.

24. The method of claim 22, wherein the configuring of the datapath is accomplished by a processor in the controller device.

25. The method of claim 22, wherein the command is associated with the data port by an explicit command tag in the data access command sent to the bridge device.

26. The method of claim 22, further comprising:

receiving, in the controller device, a completion message from the bridge device indicating completion of a command or an error message indicating an error encountered while executing the command.

27. The method of claim 21, wherein the interface is PCIe.

28. The method of claim 21, further comprising completing the following in the controller device:

in response to the bridge device sending data path program data, initiating configuration of a data path in the controller to prepare for transfer of data relating to the command in accordance with the data path program data, wherein the data path program data was previously sent by the controller device to the bridge device and is associated with the command from the controller device.

29. A method for processing data operations in a controller architecture, wherein the controller architecture comprises a controller device and a bridge device coupled with non-volatile storage, the method comprising completing the following operations in the bridge device:

receiving a plurality of data access commands from the controller device through an interface connecting the controller device and the bridge device, at least some of the data access commands including physical address instructions associated with the commands;

selecting one of the commands for processing; and

accessing a data port on the controller associated with the selected command to initiate a transfer of data for the command.

30. The method of claim 29, further comprising completing the following in the bridge device:

reporting to the controller a status of an operation performed in the non-volatile storage as a response to the selected data access command.

31. The method of claim 29, wherein each of the data access commands includes a priority indication, and wherein the selecting is based on the priority indication in the data access command and a current state of one or more data channels in the non-volatile storage.

32. The method of claim 29, wherein the interface is PCIe.