CN104049951A - Replaying memory transactions while resolving memory access faults - Google Patents

Abstract

One embodiment of the invention is a Parallel Processing Unit (PPU) comprising one or more Streaming Multiprocessors (SM) and implementing a replay unit per SM. Upon detection of a page fault associated with a memory transaction issued by a particular SM, the corresponding replay unit stalls that SM, but not any unaffected SMs, from issuing new memory transactions. The replay unit then stores the failed memory transaction and any failed in-flight memory transactions in a replay buffer. When the page fault is resolved, the replay unit replays the memory transactions in the replay buffer—removing successful memory transactions from the replay buffer—until all stored memory transactions have successfully executed. Advantageously, the overall performance of the PPU is improved compared to conventional PPUs that, upon detection of a page fault, halt execution of memory transactions across all SMs included in the PPU until the fault is resolved.

Description

Replaying memory transactions while resolving memory access failures

technical field

The present invention relates generally to computer science, and more specifically, to replaying memory transactions while resolving memory access failures.

Background technique

A typical computer system includes a central processing unit (CPU) and a parallel processing unit (PPU). When a software application executes on a computer system, the CPU and PPU perform memory operations to store and retrieve data from physical memory locations. Some advanced computer systems implement a unified virtual memory architecture (UVM) shared by the CPU and PPU. Among other things, the architecture enables the CPU and PPU to use a common (eg, same) virtual memory address to access a physical memory location, regardless of whether the physical memory location is within system memory or the PPU's local memory (PPU memory).

Computer systems typically include memory management functions to facilitate virtual memory and paging operations. During normal operation, an instruction may request access to a virtual address associated with a page of data that is paged out, causing an access fault. In response to an access failure, a conventional processing unit may complete instructions preceding the failing instruction and cancel the failing instruction along with all instructions beginning execution after the failing instruction. At this point, the access fault handler (fault handler) pages in the requested data page (page-in), and restarts execution from the faulty instruction. In some cases, accessing a fault handler may require a significant amount of time to complete relative to typical instruction execution times. In particular, if the computer system implements a unified virtual memory architecture, the access fault handler executes a lengthy faulting procedure that migrates memory pages between system and PPU-local memory.

In a highly parallel multi-threaded advanced PPU, hundreds of memory transactions and thus many address translations can be prominent at any one time. Therefore, many memory access faults are active at any time. If the PPU were to implement conventional instruction cancellation fault handling techniques, the PPU would frequently cancel thousands of instructions across all execution units. In addition, the PPU will have to wait for a lengthy access fault handler to load paged out data for each faulted instruction within each thread of execution. Such latencies will greatly and often unacceptably degrade overall system performance.

As previously stated, what is needed in the art is a more efficient method of handling access faults involving multi-threaded processing units.

Contents of the invention

One embodiment of the invention is directed to a computer-implemented method for processing virtual memory transactions associated with a multi-threaded processing unit. The method includes: receiving a first virtual memory transaction from a first unit; attempting to execute the first virtual memory transaction; detecting a first page fault associated with the first virtual memory transaction; storing the first virtual memory transaction in a replay buffer; Initiating a stall condition that prohibits the first unit from generating subsequent virtual memory transactions until the first page fault has been resolved; and once the first page fault has been resolved, re-executing the first virtual memory transaction and storing in the replay cache at least one other virtual memory transaction in the server.

One advantage of the disclosed method is that units contained in multithreaded processing units that did not contribute to a page fault can continue to issue virtual memory transactions while a page fault exists. Furthermore, since the affected cells continue to replay the failed virtual memory transaction as well as the failed in-flight virtual memory transaction, these virtual memory transactions are not canceled while the page fault is being resolved. Thus, the overall performance of a multi-threaded processing unit is improved compared to a conventional multi-threaded processing unit which cancels virtual memory transactions issued by all units within the multi-threaded processing unit as soon as a page fault occurs , until the page fault is resolved.

Description of drawings

So that the above recited features of the present invention can be understood in detail, and a more particular description of the invention, briefly summarized above, may be had by reference to exemplary embodiments, some of which are shown in the accompanying drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may have other equally effective embodiments.

Figure 1 is a block diagram illustrating a computer system configured to implement one or more aspects of the present invention;

Figure 2 is a block diagram illustrating a unified virtual memory system (UVM), according to one embodiment of the present invention;

3 is a block diagram illustrating a unified virtual memory system (UVM) configured with a replay unit, according to one embodiment of the present invention;

FIG. 4 is a conceptual diagram illustrating the replay unit of FIG. 3 according to one embodiment of the present invention; and

Figure 5 is a flowchart of method steps for managing memory transactions issued by a Streaming Multiprocessor (SM) according to one embodiment of the present invention.

Detailed ways

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without one or more of these specific details.

System Overview

FIG. 1 is a block diagram illustrating a computer system 100 configured to implement one or more aspects of the present invention. Computer system 100 includes a central processing unit (CPU) 102 and system memory 104 in communication via an interconnection path that may include a memory bridge 105 . The memory bridge 105 may be, for example, a north bridge chip, connected to an I/O (input/output) bridge 107 via a bus or other communication path 106 (eg, a HyperTransport link). I/O bridge 107 , which may be, for example, a south bridge chip, receives user input from one or more user input devices 108 (eg, keyboard, mouse) and forwards the input to CPU 102 via communication path 106 and memory bridge 105 . Parallel processing subsystem 112 is connected to memory bridge 105 via a bus or second communication path 113 such as Peripheral Component Interconnect (PCI) Express, Accelerated Graphics Port, or HyperTransport Link; in one embodiment, parallel processing subsystem 112 is the graphics subsystem that transfers the pixels to the display device 110, which may be any conventional cathode ray tube, liquid crystal display, light emitting diode display, or the like. System disk 114 may also be connected to I/O bridge 107 and may be configured to store applications and data and content used by CPU 102 and parallel processing subsystem 112 . System disk 114 provides non-volatile storage space for applications and data, and may contain fixed or removable hard drives, flash drives, and CD-ROM (Compact Disc Read-Only Memory), DVD-ROM (Digital Versatile Disc -ROM), Blu-ray, HD-DVD (high-resolution DVD), or other magnetic, optical, or solid-state storage devices.

Switch 116 provides connections between I/O bridge 107 and other components such as network adapter 118 and various add-in cards 120 and 121 . Other components (not explicitly shown), including Universal Serial Bus (USB) or other port connections, compact disc (CD) drives, digital versatile disc (DVD) drives, film recording devices, and similar components, may also be connected to the I/O O bridge 107. The various communication paths shown in Figure 1, including specifically named communication paths 106 and 113, may be implemented using any suitable protocol, such as PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol , and as known in the art, connections between different devices may use different protocols.

In one embodiment, parallel processing subsystem 112 includes circuitry optimized for graphics and video processing, including, for example, video output circuitry, and constitutes one or more parallel processing units (PPUs) 202 . In another embodiment, the parallel processing subsystem 112 includes circuitry optimized for general-purpose processing while preserving the underlying computing architecture, as described in greater detail herein. In yet another embodiment, parallel processing subsystem 112 may be integrated with one or more other system elements in a single subsystem, such as combining memory bridge 105, CPU 102, and I/O bridge 107, to form a system-on-chip (SoC) . It is well known that many graphics processing units (GPUs) are designed to perform parallel operations and calculations, and thus are considered a type of parallel processing unit (PPU).

Any number of PPUs 202 may be included in parallel processing subsystem 112 . For example, multiple PPUs 202 may be provided on a single add-in card, or multiple add-in cards may be connected to communication path 113, or one or more PPUs 202 may be integrated into a bridge chip. PPUs 202 in a multiple PPU system may be the same or different from each other. For example, different PPUs 202 may have different numbers of processing cores, different amounts of local parallel processing memory, and so on. Where multiple PPUs 202 are present, those PPUs may be operated in parallel to process data at a higher throughput than is possible with a single PPU 202 . Systems incorporating one or more PPUs 202 can be implemented in a variety of configurations and form factors, including desktop, notebook or handheld personal computers, servers, workstations, game consoles, embedded systems, and the like.

PPU 202 advantageously implements a highly parallel processing architecture. PPU 202 includes a number of general purpose processing clusters (GPCs). Each GPC is capable of concurrently executing a large number (eg, hundreds or thousands) of threads, where each thread is an instance of a program. In some embodiments, single instruction, multiple data (SIMD) instruction issue techniques are used to support parallel execution of a large number of threads without providing multiple independent instruction units. In other embodiments, single instruction, multiple thread (SIMT) technology is used to support parallel execution of a large number of generally simultaneous threads using a common instruction unit configured to issue instructions to a set of processing engines within each of the GPCs 208 . Unlike the SIMD execution mechanism where all processing engines typically execute the same instructions, SIMT execution allows different threads to more easily follow distributed execution paths through a given thread program.

A GPU includes a large number of streaming multiprocessors (SMs), where each SM is configured to process one or more thread groups. A series of instructions delivered to a particular GPC 208 constitutes a thread, and a collection of some number of concurrently executing threads across parallel processing engines (not shown) within an SM is referred to herein as a "warp" or "thread group." ". As used herein, a "thread group" refers to a group of threads that concurrently execute the same program on different input data, one thread of the group being assigned to a different processing engine within the SM. Additionally, multiple related thread groups can be active within an SM concurrently (at different stages of execution). This collection of thread groups is referred to herein as a "cooperative thread array" ("CTA") or "thread array."

In embodiments of the present invention, it may be desirable to use a computing system's PPU 202 or other processor to perform general-purpose computations using an array of threads. Each thread in the thread array is assigned a unique thread identifier ("thread ID") that is accessible to the thread during its execution. Thread IDs, which can be defined as one-dimensional or multi-dimensional values, control aspects of thread processing behavior. For example, the thread ID can be used to determine which portion of an input data set a thread is to process and/or to determine which portion of an output data set a thread is to produce or write.

In operation, CPU 102 is the main processor of computer system 100, controlling and coordinating the operation of other system components. Specifically, CPU 102 issues commands to control the operation of PPU 202 . In one embodiment, communication path 113 is a PCI Express link, as is known in the art, where a dedicated lane is assigned to each PPU 202. Other communication paths may also be used. PPU 202 advantageously implements a highly parallel processing architecture. The PPU 202 can be equipped with any amount of local parallel processing memory (PPU memory).

In some embodiments, system memory 104 includes a unified virtual memory (UVM) driver 101 . The UVM driver 101 includes instructions for performing various tasks related to the unified virtual memory (UVM) shared by the CPU 102 and the PPU 202 . Among other things, the architecture enables CPU 102 and PPU 202 to use common virtual memory addresses to access physical memory locations, whether within system memory 104 or in memory local to PPU 202 (PPU memory).

It should be understood that the systems shown herein are exemplary and that variations and modifications are possible. The connection topology, including the number and arrangement of bridges, the number of CPUs 102, and the number of parallel processing subsystems 112, can be modified as desired. For example, in some embodiments, system memory 104 is connected directly to CPU 102 rather than through a bridge, and other devices communicate with system memory 104 via memory bridge 105 and CPU 102 . In other alternative topologies, parallel processing subsystem 112 is connected to I/O bridge 107 or directly to CPU 102 instead of memory bridge 105 . Yet in other embodiments, I/O bridge 107 and memory bridge 105 may be integrated onto a single chip rather than exist as one or more discrete devices. Larger embodiments may include two or more CPUs 102 and two or more parallel processing systems 112 . Certain components shown herein are optional; for example, any number of add-in cards or peripherals may be supported. In some embodiments, switch 116 is eliminated and network adapter 118 and add-in cards 120 , 121 are directly connected to I/O bridge 107 .

Unified Virtual Memory System Architecture

FIG. 2 is a block diagram illustrating a unified virtual memory (UVM) system 200 according to one embodiment of the present invention. As shown, the unified virtual memory system 200 includes, without limitation, a CPU 102 , a system memory 104 , and a parallel processing unit (PPU) 202 connected to a parallel processing unit memory (PPU memory) 204 . CPU 102 and system memory 104 are connected to each other and to PPU 202 via memory bridge 105 .

CPU 102 executes threads that may request data stored in system memory 104 or PPU memory 204 via virtual memory addresses. From knowledge of the inner workings of the memory system, virtual memory addresses shield threads that are executing in CPU 102 . Thus, a thread may only know the virtual memory address, and may access data by requesting the data through the virtual memory address.

CPU 102 includes CPU MMU 209, which handles requests from CPU 102 for translating virtual memory addresses into physical memory addresses. Accessing data stored in physical memory units such as system memory 104 and PPU memory 204 requires physical memory addresses. CPU 102 includes CPU fault handler 211 that performs steps to make requested data available to CPU 102 in response to CPUMMU 209 generating a page fault. The CPU fault handling program 211 is generally software that resides in the system memory 104 and executes on the CPU 102 , and is woken up by an interrupt to the CPU 102 .

System memory 104 stores various memory pages (not shown) used by threads executing on CPU 102 or PPU 202 . As shown, system memory 104 stores CPU page tables 206, which contain mappings between virtual memory addresses and physical memory addresses. System memory 104 also stores page state directory 210, which acts as a "home table" for UVM system 200, as discussed in more detail below. System memory 104 stores fault buffer 216 containing entries written by PPU 202 to notify CPU 102P of page faults generated by PU 202 . In some embodiments, system memory 104 includes a unified virtual memory (UVM) driver 101 that contains instructions that, when executed, cause CPU 102 to, among other things, execute commands for repairing page faults. In alternative embodiments, any combination of page state directory 210 and one or more command queues 214 may be stored in PPU memory 204 . Additionally, PPU page tables 208 may be stored in system memory 104 .

In a similar manner to CPU 102 , PPU 202 executes instructions that may request data stored in system memory 104 or PPU memory 204 via virtual memory addresses. PPU 202 includes PPU MMU 213, which handles requests from PPU 202 for translating virtual memory addresses into physical memory addresses. PPU 202 also includes copy engine 212 that executes commands stored in command queue 214 for copying memory pages, modifying data in PPU page table 208, and other commands. PPU fault handler 215 performs steps in response to a page fault on PPU 202 . PPU fault handler 215 may be software running on a processor or a dedicated microcontroller in PPU 202 . Alternatively, PPU fault handler 215 may be a combination of software running on CPU 102 and software running on a dedicated microcontroller in PPU 202 in communication with each other. In some embodiments, CPU fault handler 211 and PPU fault handler 215 may be invoked by a fault on either CPU 102 or PPU 202 . Command queue 214 may be in PPU memory 204 or system memory 104 , but is preferably located in system memory 104 .

In some embodiments, CPU fault handler 211 and UVM driver 101 may be a unified software program. In such cases, the unified software program may be software that resides in system memory 104 and executes on CPU 102 . PPU fault handler 215 may be a separate software program running on a processor or a dedicated microcontroller in PPU 202 , or PPU fault handler 215 may be a separate software program running on CPU 102 .

In other embodiments, PPU fault handler 215 and UVM driver 101 may be a unified software program. In such cases, the unified software program may be software that resides in system memory 104 and executes on CPU 102 . CPU fault handler 211 may be software that resides in system memory 104 and executes on CPU 102 .

In other embodiments, the CPU fault handler 211, the PPU fault handler 215, and the UVM driver 101 may be a unified software program. In such cases, the unified software program may be software that resides in system memory 104 and executes on CPU 102 .

In some embodiments, CPU fault handler 211 , PPU fault handler 215 , and UVM driver 101 may all reside in system memory 104 as described above. As shown in FIG. 2 , UVM driver 101 resides in system memory 104 , while CPU fault handler 211 and PPU fault handler 215 reside in CPU 102 .

CPU fault handler 211 and PPU fault handler 215 may respond to hardware interrupts emanate from CPU 102 or PPU 202 , such as interrupts due to page faults. As described further below, UVM driver 101 contains instructions for performing various tasks related to the management of UVM system 200, including without limitation repairing page faults and accessing CPU page table 206, page state directory 210, and/or fault cache device 216.

In some embodiments, CPU page table 206 and PPU page table 208 have different formats and contain different information; for example, PPU page table 208 may contain the following information while CPU page table 206 does not: atomic disable bit (atomic disable bit); compression tag; and memory swizzling type.

In a similar manner to system memory 104 , PPU memory 204 stores various pages (not shown). As shown, PPU memory 204 also includes a PPU page table 208 that contains a mapping between virtual memory addresses and physical memory addresses. Alternatively, PPU page table 208 may be stored in system memory 104 .

translate virtual memory address

When a thread executing in the CPU 102 requests data via a virtual memory address, the CPU 102 requests translation of the virtual memory address into a physical memory address from the CPU memory management unit (CPU MMU) 209 . In response, CPU MMU 209 attempts to translate the virtual memory address into a physical memory address specifying a location in a memory unit, such as system memory 104, where the data requested by CPU 102 is stored.

To translate a virtual memory address to a physical memory address, CPU MMU 209 performs a lookup operation to determine whether CPU page table 206 contains a mapping associated with the virtual memory address. In addition to virtual memory addresses, requests to access data may also specify a virtual memory address space. The unified virtual memory system 200 can implement multiple virtual memory address spaces, each space is assigned one or more threads. Virtual memory addresses are unique within any given virtual memory address space. Furthermore, virtual memory addresses within a given virtual memory address are contiguous across CPU 102 and PPU 202 , allowing the same virtual memory address to point to the same data across CPU 102 and PPU 202 . In some embodiments, two virtual memory addresses may point to the same data, but may not map to the same physical memory address (eg, CPU 102 and PPU 202 may each have a local read-only copy of the data).

For any given virtual memory address, CPU page table 206 may or may not contain a mapping between the virtual memory address and the physical memory address. If CPU page table 206 contains a mapping, CPU MMU 209 reads the mapping to determine the physical memory address associated with the virtual memory address and provides the physical memory address to CPU 102. However, if CPU page table 206 does not contain a mapping associated with the virtual memory address, then CPU MMU 209 cannot translate the virtual memory address into a physical memory address, and CPU MMU 209 generates a page fault. To repair the page fault and make the requested data available to CPU 102, a "page fault sequence" is executed. More specifically, CPU 102 reads PSD 210 to find out the current mapping state of the page and then determines the appropriate page fault sequence. The page fault sequence typically maps the memory page associated with the requested virtual memory address or changes the type of access granted (eg, read access, write access, atomic access). The different types of page fault sequences implemented in UVM system 200 are discussed in more detail below.

Within UVM system 200, data associated with a given virtual memory address may be stored in system memory 104, PPU memory 204, or both system memory 104 and PPU memory 204 as a read-only copy of the same data. Furthermore, for any such data, either or both CPU page table 206 and PPU page table 208 may contain a mapping associated with the data. Note that there is some data for which a mapping exists in one page table but not another. However, PSD 210 contains all mappings stored in PPU page table 208 as well as PPU-related mappings stored in CPU page table 206 . PSD 210 thus serves as the “master” page table for unified virtual memory system 200 . Thus, when CPU MMU 209 does not find a mapping in CPU page table 206 associated with a particular virtual memory address, CPU 102 reads PSD 210 to determine whether PSD 210 contains a mapping associated with that virtual memory address. Various embodiments of PSD 210 may contain different types of information associated with virtual memory addresses in addition to the mappings associated with virtual memory addresses.

When the CPU MMU 209 generates a page fault, the CPU fault handler 211 performs a series of operations for the appropriate page fault sequence to repair the page fault. Also, during the page fault sequence, CPU 102 reads PSD 210 and performs additional operations to change mappings or permissions within CPU page table 206 and PPU page table 208 . Such operations may include reading and/or modifying CPU page tables 206, reading and/or modifying page state directories 210, and/or migrating between memory units (e.g., system memory 104 and PPU memory 204) are referred to as " memory page" block of data.

To determine which operations are to be performed in the page fault sequence, CPU 102 identifies the memory page associated with the virtual memory address. CPU 102 then reads status information about the memory page from PSD 210 associated with the virtual memory address associated with the memory access request that caused the page fault. Such state information may additionally include an ownership state regarding the memory page associated with the virtual memory address. For any given memory page, several ownership states are possible. For example, a memory page may be "CPU-owned", "PPU-owned", or "CPU-shared". A memory page is considered owned by a CPU if the CPU 102 can access the memory page via a virtual address without incurring a page fault, and if the PPU 202 cannot access the memory page via a virtual address without incurring a page fault. Preferably, all pages of the CPU reside in system memory 104 , but may also reside in PPU memory 204 . A memory page is considered owned by the PPU if the PPU 202 can access the page via the virtual address, and if the CPU 102 cannot access the page via the virtual address without incurring a page fault. Preferably, all pages of the PPU reside in PPU memory 204, but may also reside in system memory 104 when migration from system memory 104 to PPU memory 204 is not performed. Finally, memory is considered CPU shared if CPU 102 and PPU 202 can access a memory page via a virtual address without causing a page fault. CPU shared pages may reside in either system memory 104 or PPU memory 204 .

CPU page table 206 may assign an ownership state to a memory page based on various factors, including the usage history of the memory page. The usage history may include information about whether CPU 102 or PPU 202 has recently accessed a page of memory and how many times such access was made. For example, if based on the usage history of a given memory page, the UVM system 200 determines that the memory page is likely to be used primarily or exclusively by the CPU 102, the UVM system 200 can assign the memory page an ownership status of "CPU Owned" and assign the memory page Pages are placed in system memory 104 . Similarly, if based on the usage history of a given memory page, UVM system 200 determines that the memory page is likely to be used primarily or exclusively by PPU 202, UVM system 200 may assign the memory page an ownership status of "PPU Owned" and assign This page is placed in PPU memory 204 . Finally, if based on the usage history of a given memory page, UVM system 200 determines that the memory page is likely to be used by both CPU 102 and PPU 202, and determines that migrating the memory page from system memory 104 to PPU memory 204 and back would be too costly time, the UVM system 200 may assign the memory page an ownership status of "CPU Shared".

As an example, fault handlers 211 and 215 may implement any or all of the following heuristics for migration:

(a) with respect to CPU 102 accesses to unmapped (unmapped) pages that are mapped to PPU 202 and have not been migrated recently, unmap the faulty page from PPU 202, migrate the page to CPU 102, and map the page to CPU 102;

(b) with respect to PPU 202 accesses to unmapped pages that are mapped to CPU 102 and have not been migrated recently, unmap the faulty page from CPU 102, migrate the page to PPU 202, and map the page to PPU 202;

(c) with respect to CPU 102 accesses to the most recently migrated unmapped page mapped to PPU 202 , migrate the faulted page to CPU 102 and map the page on both CPU 102 and PPU 202 ;

(d) for a PPU 202 access to a most recently migrated unmapped page mapped on the CPU 102, map the page to both the CPU 102 and the PPU 202;

(e) with respect to a PPU 202 atomic access to a page mapped to both CPU 102 and PPU 202 but not enabled for atomic operations by PPU 202, unmap the page from CPU 102 and map to PPU 202 with atomic operations enabled;

(f) For PPU 202 write access to a page mapped on CPU 102 and PPU 202 as copy-on-write (COW), copy the page to PPU 202, thereby making an independent copy of the page, set The new page is mapped on the PPU as a read-write (read-write), and the current page is reserved on the CPU102;

(g) For PPU 202 read accesses to pages mapped on CPU 102 and PPU 202 as zero-fill-on-demand (ZFOD), a physical memory page on PPU 202 is allocated and filled with zeros, and the The page is mapped on the PPU, but changed to be unmapped on the CPU102;

(h) with respect to an access by the first PPU 202(1) to an unmapped page that is mapped on the second PPU 202(2) and has not been migrated recently, unmap the failed page from the second PPU 202(2), migrating the page to the first PPU 202(1), and mapping the page to the first PPU 202(1); and

(i) With respect to an access by the first PPU 202(1) to an unmapped page that was mapped on the second PPU 202(2) and was most recently migrated, map the failed page to the first PPU 202(1), and keep The mapping of this page on the second PPU 202(2).

In conclusion, many heuristic rules are possible, and the scope of the invention is not limited to these examples.

Also, any migration heuristic can be "rounded up" to include more pages or larger page sizes, for example:

(j) with respect to CPU 102 accesses to unmapped pages that are mapped to PPU 202 and have not been migrated recently, unmap the failed page plus additional pages adjacent to the failed page in virtual address space from PPU 202, Migrate these pages to CPU 102 and map these pages to CPU 102 (in a more detailed example: for a 4kB fault page, migrate the aligned 64kB region containing the 4kB fault page);

(k) with respect to PPU 202 accesses to unmapped pages that are mapped to CPU 102 and have not been migrated recently, unmap the faulted page plus additional pages adjacent to the faulted page in virtual address space from CPU 102, migrate these pages to PPU 202, and map these pages to PPU 202 (in a more detailed example: for a 4kB fault page, migrate the aligned 64kB region containing the 4kB fault page);

(l) with respect to CPU 102 accesses to unmapped pages that are mapped to PPU 202 and have not been migrated recently, unmap the failed page plus additional pages adjacent to the failed page in virtual address space from PPU 202, Migrate these pages to CPU102, map these pages to CPU102, and treat all migrated pages as one or more larger pages on CPU102 (in a more detailed example: for 4kB faulted pages, migration contains 4kB faults the aligned 64kB region of the page, and treat the aligned 64kB region as a 64kB page);

(m) with respect to PPU 202 accesses to unmapped pages that are mapped to CPU 102 and have not been migrated recently, unmap the faulted page plus additional pages adjacent to the faulted page in virtual address space from CPU 102 , Migrate these pages to PPU202, map these pages to PPU202, and treat all migrated pages as one or more larger pages on PPU202 (in a more detailed example: for 4kB faulted pages, migration contains 4kB faults the aligned 64kB region of the page, and treat the aligned 64kB region as a 64kB page);

(n) For an access by the first PPU 202(1) to an unmapped page that is mapped to the second PPU 202(2) and has not been migrated recently, append the faulted page in the virtual address space with the faulted page unmapping additional pages adjacent to the page from the second PPU 202(2), migrating these pages to the first PPU 202(1), and mapping these pages to the first PPU 202(1); and

(o) For an access by the first PPU 202(1) to a recently migrated unmapped page mapped on the second PPU 202(2), append the faulted page in the virtual address space with the faulted page Additional pages adjacent to the page are mapped to the first PPU 202(1), and the mapping of these pages on the second PPU 202(2) is maintained.

In conclusion, many heuristic rules are possible, and the scope of the invention is not limited to these examples.

In some embodiments, PSD entries may contain transitional state information to ensure proper synchronization between various requests made by units within CPU 102 and PPU 202 . For example, a PSD 210 entry may contain transition state information that a particular page is in the process of transitioning from CPU ownership to PPU ownership. Various units in CPU 102 and PPU 202, such as CPU fault handler 211 and PPU fault handler 215, upon determining that a page is in such a transitional state, may forego a partial page fault sequence to avoid a previous fault to the same virtual memory A step in the page fault sequence triggered by a virtual memory access to an address. As a specific example, if a page fault causes a page to be migrated from system memory 104 to PPU memory 204, a different page fault is detected that would cause the same migration and not cause another page migration. Additionally, various units within CPU 102 and PPU 202 may implement atomic operations for proper sequencing of operations on PSD 210 . For example, with respect to modifications to PSD 210 entries, CPU fault handler 211 or PPU fault handler 215 may issue an atomic compare and swap (swap) operation to modify the page state of a particular entry in PSD 210 . Therefore, the modification can be done without interference from the operation of other units.

Multiple PSDs 210 may be stored in system memory 104 - one for each virtual address space. A memory access request generated by either CPU 102 or PPU 202 may thus contain a virtual memory address and also identify a virtual memory address space associated with the virtual memory address.

Just as CPU 102 may execute memory access requests that include virtual memory addresses (ie, instructions that include requests to access data via virtual memory addresses), PPU 202 may execute similar types of memory access requests. More specifically, PPU 202 includes multiple execution units, such as GPCs and SMs, configured to execute multiple threads and thread groups, as described above in connection with FIG. 1 . In operation, those threads may request data from memory (eg, system memory 104 or PPU memory 204 ) by specifying a virtual memory address. Like CPU 102 and CPU MMU 209 , PPU 202 includes a PPU memory management unit (MMU) 213 . PPU MMU 213 receives requests from PPU 202 for virtual memory address translations and attempts to provide translations from PPU page tables 208 for virtual memory addresses.

Similar to CPU page table 206, PPU page table 208 contains a mapping between virtual memory addresses and physical memory addresses. As is the case with CPU page table 206, for any given virtual address, PPU page table 208 may contain no page table entries that map virtual memory addresses to physical memory addresses. Like the CPU MMU 209, when the PPU MMU 213 requests a translation of a virtual memory address from the PPU page table 208 and there is no mapping in the PPU page table 208 or the access type is not allowed by the PPU page table 208, the PPU MMU 213 generates a page fault . Subsequently, the PPU fault handler 215 triggers a page fault sequence. Also, the different types of page fault sequences implemented in UVM system 200 will be described in more detail below.

During a page fault sequence, CPU 102 or PPU 202 may write commands to command queue 214 for execution by replication engine 212 . This approach frees the CPU 102 or PPU 202 to perform other tasks while the replication engine 212 reads and executes commands stored in the command queue 214, and allows all commands for the fault sequence to be queued at the same time, thereby avoiding the progress monitoring. Commands executed by replication engine 212 may additionally include deleting, creating, or modifying page table entries in PPU page table 208 , reading or writing data from system memory 104 , and reading or writing data to PPU memory 204 .

Fault buffer 216 stores fault buffer entries specifying information related to page faults generated by PPU 202 . The fault buffer entry may include, for example, the type of access attempted (e.g., read, write, or atomic), the virtual memory address for which the attempted access caused the page fault, the virtual address space, and the reference to the page that caused the fault. An indication of the failing apartment or thread. In operation, when PPU 202 causes a page fault, PPU 202 may write a fault buffer entry into fault buffer 216 to notify PPU fault handler 215 of the faulted page and the type of access that caused the fault. PPU fault handler 215 then performs actions to repair the page fault. Because PPU 202 is executing multiple threads, fault buffer 216 may store multiple faults, each of which may cause one or more faults due to the pipelined nature of PPU 202's memory accesses.

page fault sequence

As described above, in response to receiving a request for translation of a virtual memory address, CPU MMU 209 generates a page if CPU page table 206 does not contain a mapping associated with the requested virtual memory address or does not grant the type of access being requested. Fault. Similarly, in response to receiving a request for translation of a virtual memory address, PPU MMU 213 generates a page fault if PPU page table 208 does not contain a mapping associated with the requested virtual memory address or does not grant the type of access being requested . When the CPU MMU 209 or the PPU MMU 213 generates a page fault, the thread that requested the data at the virtual memory address stalls, and the "local fault handler"—the CPU fault handler 211 for the CPU 102 or the PPU for the PPU 202 Fault Handler 215 - Attempts to repair page faults by performing a "page fault sequence". As noted above, a page fault sequence includes a series of operations that enable the faulted unit (ie, the unit that caused the page fault—either CPU 102 or PPU 202 ) to access data associated with a virtual memory address. After the page fault sequence ends, the thread that requested the data via the virtual memory address continues to execute. In some embodiments, fault recovery is simplified by allowing the fault recovery logic to track the failing memory access as opposed to the failing instruction.

If there are any changes in ownership status or changes in access permissions that a memory page associated with a page fault has to undergo, the operations performed during the page fault sequence depend on these changes. A transition from a current ownership state to a new ownership state or a change in access permissions may be part of a page fault sequence. In some examples, migrating memory pages associated with page faults from system memory 104 to PPU memory 204 is also part of the page fault sequence. In other examples, migrating memory pages associated with page faults from PPU memory 204 to system memory 104 is also part of the page fault sequence. Various heuristics described more fully herein may be used to configure UVM system 200 to change memory page ownership states or to migrate memory pages according to various sets of operating conditions and patterns. Described in more detail below is the page fault sequence for the following four memory page ownership state transitions: CPU owned to CPU shared, CPU owned to PPU owned, PPU owned to CPU owned, and PPU owned to CPU shared.

A fault generated by PPU 202 may initiate the transition from CPU ownership to CPU sharing. Prior to this transition, a thread executing in PPU 202 attempted to access data at a virtual memory address that was not mapped in PPU page table 208 . This access attempts to cause a PPU-based page fault, which then causes a fault buffer entry to be written to fault buffer 216 . In response, PPU fault handler 215 reads the PSD 210 entry corresponding to the virtual memory address and identifies the memory page associated with the virtual memory address. After reading PSD 210, PPU fault handler 215 determines that the current ownership state of the memory page associated with the virtual memory address is owned by the CPU. Based on the current ownership state and other factors, such as regarding the usage characteristics of the memory page or the type of memory access, the PPU fault handler 215 determines that the new ownership state for the page should be CPU shared.

To change ownership status, PPU fault handler 215 writes a new entry in PPU page table 208 corresponding to the virtual memory address and associating the virtual memory address with the memory page identified via the PSD 210 entry. PPU fault handler 215 also modifies the PSD 210 entry for the memory page to indicate that the ownership status is CPU shared. In some embodiments, a translation look-aside buffer (TLB) in the PPU 202 is invalidated to account for cases where translations to invalid pages are cached. At this point, the page fault sequence ends. The ownership status of the memory page is CPU shared, meaning that the memory page is accessible to both CPU 102 and PPU 202 . Both CPU page table 206 and PPU page table 208 contain entries that associate virtual memory addresses to memory pages.

A fault generated by PPU 202 may initiate a transition from CPU ownership to PPU ownership. Prior to this transition, an operation being executed in PPU 202 attempted to access data at a virtual memory address that was not mapped in PPU page table 208 . This memory access attempts to cause a PPU-based page fault, which then causes a fault buffer entry to be written to fault buffer 216 . In response, PPU fault handler 215 reads the PSD 210 entry corresponding to the virtual memory address and identifies the memory page associated with the virtual memory address. After reading PSD 210, PPU fault handler 215 determines that the current ownership state of the memory page associated with the virtual memory address is owned by the CPU. Based on the current ownership state and other factors, such as regarding the usage characteristics of the page or the type of memory access, the PPU fault handler 215 determines that the new ownership state for the page should be owned by the PPU.

PPU 202 writes a fault buffer entry into fault buffer 216 indicating that PPU 202 generated a page fault and designating the virtual memory address associated with the page fault. PPU fault handler 215 executing on CPU 102 reads the fault register entry, and in response CPU 102 removes the mapping in CPU page table 206 associated with the virtual memory address that caused the page fault. CPU 102 may flush the cache before and/or after removing the mapping. CPU 102 also writes a command into command queue 214 instructing PPU 202 to copy a page from system memory 104 into PPU memory 204 . Copy engine 212 in PPU 202 reads commands in command queue 214 and copies pages from system memory 104 to PPU memory 204 . PPU 202 writes a page table entry into PPU page table 208 corresponding to the virtual memory address and associates the virtual memory address with the newly copied memory page in PPU memory 204 . Writing to PPU page table 208 may be done via PPU 202 . Alternatively, CPU 102 may update PPU page table 208 . The PPU fault handler 215 also modifies the PSD 210 for the memory page to indicate that the ownership status is owned by the PPU. In some embodiments, entries in the TLB in PPU 202 or CPU 102 may be invalidated to account for cases where translations are cached. At this point, the page fault sequence ends. The ownership status of the memory page is owned by the PPU, meaning that the memory page is only accessible to the PPU 202 . Only PPU page table 208 contains an entry associating a virtual memory address with that memory page.

A fault generated by CPU 102 may initiate a transition from PPU ownership to CPU ownership. Prior to this transition, an operation being executed in CPU 102 attempted to access data at a virtual memory address that was not mapped in CPU page table 206, causing a CPU-based page fault. CPU fault handler 211 reads the PSD 210 entry corresponding to the virtual memory address and identifies the memory page associated with the virtual memory address. After reading PSD 210, CPU fault handler 211 determines that the current ownership status of the memory page associated with the virtual memory address is owned by the PPU. Based on the current ownership state and other factors, such as regarding the usage characteristics of the page or the type of access, the CPU fault handler 211 determines that the new ownership state of the page is owned by the CPU.

CPU fault handler 211 changes the ownership state associated with the memory page to CPU owned. CPU fault handler 211 writes a command into command queue 214 to cause replication engine 212 to remove from PPU page table 208 an entry associating a virtual memory address with the memory page. Various TLB entries can be invalidated. CPU fault handler 211 also copies memory pages from PPU memory 204 into system memory 104 , which may be done via command queue 214 and copy engine 212 . CPU fault handler 211 writes a page table entry in CPU page table 206 that associates a virtual memory address with a memory page copied into system memory 104 . CPU fault handler 211 also updates PSD 210 to associate virtual memory addresses with newly copied memory pages. At this point, the page fault sequence ends. The ownership status of the memory page is CPU, meaning that the memory page is only accessible to the CPU 102 . Only CPU page table 206 contains an entry associating a virtual memory address with that memory page.

A fault generated by CPU 102 may initiate the transition from PPU ownership to CPU sharing. Prior to this transition, an operation being executed in CPU 102 attempted to access data at a virtual memory address that was not mapped in CPU page table 206, causing a CPU-based page fault. CPU fault handler 211 reads the PSD 210 entry corresponding to the virtual memory address and identifies the memory page associated with the virtual memory address. After reading PSD 210, CPU fault handler 211 determines that the current ownership status of the memory page associated with the virtual memory address is owned by the PPU. Based on the current ownership state and other factors, such as regarding the usage characteristics of the page, the CPU fault handler 211 determines that the new ownership state of the page is CPU shared.

The CPU fault handler 211 changes the ownership state associated with the memory page to CPU shared. CPU fault handler 211 writes a command into command queue 214 to cause replication engine 212 to remove from PPU page table 208 an entry associating a virtual memory address with the memory page. Various TLB entries can be invalidated. CPU fault handler 211 also copies memory pages from PPU memory 204 into system memory 104 . This copy operation can be accomplished via command queue 214 and copy engine 212 . CPU fault handler 211 then writes a command into command queue 214 to cause replication engine 212 to change an entry in PPU page table 208 so that a virtual memory address is associated with a memory page in system memory 104 . CPU fault handler 211 writes a page table entry to CPU page table 206 to associate a virtual memory address with a memory page in system memory 104 . CPU fault handler 211 also updates PSD 210 to associate virtual memory addresses with memory pages in system memory 104 . At this point, the page fault sequence ends. The ownership status of the page is CPU shared, and the memory page has been copied into system memory 104 . Because CPU page table 206 contains an entry associating a virtual memory address with a memory page in system memory 104 , the page is accessible to CPU 102 . Since PPU page table 208 contains an entry associating a virtual memory address with a memory page in system memory 104 , that page is also accessible to PPU 202 .

Detailed example of a page fault sequence

In this context, a detailed description of the page fault sequence performed by the PPU fault handler 215 in the event of a transition from CPPU-owned to CPU-shared is now provided to show how atomic operations and transition states are used to manage the sequence more efficiently. The page fault sequence is triggered by a PPU 202 thread attempting to access a virtual address for which no associated mapping exists in the PPU page table 208 . When a thread attempts to access data via a virtual memory address, the PPU 202 (specifically, a user-level thread) requests a translation from the PPU page table 208 . In response, a PPU page fault occurs because the PPU page table 208 does not contain a mapping associated with the requested virtual memory address.

After a page fault occurs, the aforementioned threads are trapped, stalled, and the PPU fault handler 215 executes the page fault sequence. PPU fault handler 215 reads PSD 210 to determine which memory page is associated with the virtual memory address and to determine the status of the virtual memory address. PPU fault handler 215 determines from PSD 210 that the ownership status of the memory page is CPU owned. Thus, data requested by PPU 202 is accessible to PPU 202 via the virtual memory address. The status information for the memory page also indicates that the requested data cannot be migrated to PPU memory 204 .

Based on the state information obtained from PSD 210, PPU fault handler 215 determines that the new state of the memory page should be CPU shared. PPU Fault Handler 215 changes state to "Transition to CPU Sharing". This status indicates that the page is currently in the process of transitioning to CPU sharing. When the PPU fault handler 215 is running on the microcontroller in the memory management unit, then the two processors will asynchronously update the PSD 210, using an atomic compare-swap (“CAS”) operation on the PSD 210 to change state to “Transition To GPU visible (visible)" (CPU sharing).

PPU 202 updates PPU page table 208 to associate virtual memory addresses with memory pages. PPU 202 also invalidates TLB cache entries. Next, PPU 202 performs another atomic compare-swap on PSD 210 to change the ownership state associated with the memory page to CPU shared. Finally, the page fault sequence terminates, and the thread that requested data via the virtual memory address continues to execute.

Variation of UVM system architecture

Various modifications to unified virtual memory system 200 are possible. For example, in some embodiments, upon writing a fault register entry in fault register 216, PPU 202 may trigger a CPU interrupt to cause CPU 102 to read the fault register entry in fault register 216 and respond to the fault register entry and perform any appropriate action. In other embodiments, the CPU 102 may periodically poll the fault register 216 . If CPU 102 finds a faulty register entry in faulty register 216, CPU 102 performs a series of operations in response to the faulty register entry.

In some embodiments, system memory 104 , rather than PPU memory 204 , stores PPU page table 208 . In other embodiments, a single-level or multi-level cache hierarchy, such as a single-level or multi-level translation lookaside buffer (TLB) hierarchy (not shown), may be implemented for the CPU page table 206 or the PPU Page table 208 caches virtual address translations.

In still other embodiments, PPU 202 may take one or more actions in the event a thread executing in PPU 202 causes a PPU fault ("faulted thread"). These actions include stalling the entire PPU 202, stalling the SM executing the failing thread, stalling the PPU MMU 213, stalling only the failing thread, or stalling one or more levels of the TLB. In some embodiments, after a PPU page fault occurs, and the unified virtual memory system 200 has executed the page fault sequence, the faulted thread continues to execute, and the faulted thread retries the memory access request that caused the page fault. In some embodiments, stalls at the TLB are performed in such a way that they appear as long-latency memory accesses to the faulting SM or thread, thereby requiring the SM to do nothing in response to the fault. special operations.

Finally, in other alternative embodiments, UVM driver 101 may contain instructions that cause CPU 102 to perform one or more operations for managing UVM system 200 and repairing page faults, such as accessing CPU page tables 206, PSD 210, and/or fault buffers 216. In other embodiments, an operating system kernel (not shown) may be configured to manage UVM system 200 and repair page faults by accessing CPU page table 206 , PSD 210 , and/or fault buffer 216 . In yet other embodiments, an operating system kernel may operate in conjunction with UVM driver 101 to manage UVM system 200 and repair page faults by accessing CPU page table 206 , PSD 210 and/or fault buffer 216 .

Stall and replay glitches

As noted above, UVM system 200 typically relies at least in part on CPU 102 to repair memory access faults (ie, page faults) generated by PPU 202 . In the event of a memory access failure, a conventional PPU cancels the failed memory transaction along with all memory transactions within the PPU that began executing after the failed memory transaction. SMs in such conventional PPUs do not continue to issue memory transactions until the memory access fault is resolved. Instead, to reduce the overall performance degradation associated with a failed memory transaction, the PPU 202 is configured to stall only the SM that issued the failed memory transaction. While the SM is stalled, the PPU 202 executes "in-flight" memory transactions that the SM issued prior to the failed memory transaction. In addition, the SM continues to replay the failed memory transaction and any in-flight memory transactions that did not complete successfully until all of these memory transactions succeed. Advantageously, SMs that did not cause any outstanding memory access faults continue to issue memory transactions, and the PPU 202 continues to execute these memory transactions while the UVM system 200 repairs outstanding memory access faults.

In general, the techniques described herein are illustrative rather than restrictive, and can be modified to reflect various implementations without departing from the broader spirit and scope of the invention. For example, an SM is one of many units that can issue memory transactions. Embodiments of the current invention may include any number and type of execution units in place of or in combination with SMs. Furthermore, selectively stopping only specific units within the PPU and repairing memory transactions while resolving memory access failures can be implemented in any technically flexible manner. For example, the PPU may replay failed memory transactions issued by certain "replayable" units and discard failed memory transactions issued by other units.

The selective stall and replay functions described herein can be implemented in the PPU MMU 213, a different memory management unit, a dedicated hardware unit, or software executing on a programmable hardware unit—in any combination. Furthermore, PPU 202 may be included in any type of computer system. For example, PPU 202 may be included in a computer system that does not implement a unified virtual memory architecture.

PPU with replay unit

FIG. 3 is a block diagram illustrating a unified virtual memory system (UVM) 200 configured with a replay unit 350 according to one embodiment of the present invention. PPU 202 includes any number N of streaming multiprocessors (SM) 310 and N replay units 350 - one replay unit 350 per SM 310 . For example, if the PPU 202 is to include 32 SMs 310 (0:31), the PPU 202 will include 32 replay units 350 (0:31). Each replay unit 350 enables the PPU 202 to stall the corresponding SM 310 while replaying selected memory transactions without delaying other SMs 310 .

FIG. 4 is a conceptual diagram illustrating the replay unit 350(0) of FIG. 3, according to one embodiment of the present invention. As shown, the replay unit 350(0) includes, without limitation, a transaction multiplexer (transaction Mux) 420, a micro-translation lookaside buffer (uTLB) 430, an in-flight buffer 440, a fault detector 450, and a replay buffer device 460.

Typically, threads executing within SM 310(0) each generate a virtual memory transaction stream from SM 310(0). After SM 310 ( 0 ) issues a particular virtual memory transaction from SM 310 ( 0 ), the virtual memory transaction from SM 310 ( 0 ) passes through transaction Mux 420 before reaching uTLB 430 and in-flight buffer 440 .

uTLB 430 performs one or more lookup operations to map the virtual memory address of the virtual memory transaction from SM 310(0) to a physical memory address in PPU memory 204 . Note that uTLB 430 is configured to cache mappings, which are further represented by the hierarchy of TLB caches. A page table or global TLB data structure (not shown) is configured to store all mappings across all virtual address spaces associated with a processor complex (complex) including one or more PPUs 202 and one or more CPUs 102 .

Those skilled in the art will recognize that lookup operations performed by uTLB 430 can be time consuming in the event of a cache miss. Accordingly, in-flight buffer 440 queues virtual memory transactions from SM 310(0) in first-in-first-out order, thereby maintaining the context of virtual memory transactions from SM 310(0) relative to uTLB 430 lookup operations.

If the uTLB successfully processes the virtual memory transaction from SM 310 ( 0 ), fault detector 450 routes the corresponding physical memory transaction to PPU memory 204 . In the physical memory transaction to the PPU memory 204 , the virtual address included in the virtual memory transaction from SM 310 ( 0 ) is replaced with the physical address obtained by the lookup operation of uTLB 430 .

Conversely, if uTLB 430 cannot map the virtual address specified by the virtual memory transaction from SM 310 ( 0 ), or if the virtual address needs to change the disposition of the target page of memory, uTLB 430 generates a memory access fault. Fault detector 450 handles memory access faults - sends a fault signal to CPU 102 and temporarily disables SM 310(0) from issuing new virtual memory transactions. Advantageously, fault detector 450 does not stop any other SM 310 included in PPU 202 from issuing new virtual memory transactions.

As part of handling virtual memory faults, fault detector 450 causes fault register entries to be written to fault register 216 of FIG. 2 . Also failure detector 450 performs a write operation, storing the failed virtual memory transaction from SM 310 ( 0 ) in replay buffer 460 . Additionally, fault detector 450 causes any virtual memory transactions from SM 310(0) queued in in-flight buffer 440 to complete execution. If any of these virtual memory transactions also fail, then failure detector 450 performs a write operation, storing the additional failed virtual memory transaction in replay buffer 460 . Optionally, but preferably, fault detector 450 causes fault buffer entries corresponding to additional faulted virtual memory transactions to be written to fault buffer 216 .

The PPU fault handler 215 then executes a page fault sequence designed to resolve memory access faults. Once the one or more memory access failures are resolved, CPU 102 sends a replay signal to replay unit 350(0). The CPU 102 can generate the replay signal at any time and in any technically flexible manner. Preferably, a PPU fault handler 215 included in CPU 102 generates a replay signal, typically via command queue 214 . In this way, access fault resolvers that have high overhead costs can be executed together, improving overall performance. Generating the replay signal via the command queue 214 also allows the replay operation to be synchronized with the resolve fault command, pipelining the fault resolve operation and the replay operation, thereby allowing the PPU fault handler 215 to "fire-and-forget" way to operate. In alternative embodiments, CPU 102 or PPU 202 may generate the replay signal in any technically flexible manner. For example, PPU 202 may generate rebroadcast signals at predetermined time intervals, resulting in periodic rebroadcasts at a fixed frequency.

Upon receipt of the replay signal, replay unit 350(0) invalidates uTLB 430 and transaction Mux 420 routes failed virtual memory transactions in replay buffer 460 to uTLB 430 . For each of these failed virtual memory transactions, uTLB 430 attempts to map a virtual memory address to an accessible physical memory address. If uTLB 430 successfully maps a virtual memory transaction contained in replay buffer 460 , fault detector 450 routes the corresponding physical memory transaction to PPU memory 204 . However, if uTLB 430 is unable to map a particular virtual memory transaction contained in replay buffer 460 , fault detector 450 performs a write operation to re-request the virtual memory transaction in replay buffer 460 . Note that as a result of CPU 102 successfully repairing each particular page fault, the corresponding virtual memory transaction succeeds, a physical memory transaction occurs, and the virtual memory transaction is removed from replay buffer 460 .

Replay unit 350(0) continues to re-execute the memory transactions contained in replay buffer 460 until replay buffer 460 is empty. After replay unit 350(0) determines that replay buffer 460 is empty, replay unit 350 instructs SM 310(0) to continue issuing virtual memory transactions from SM 310(0). After SM 310(0) continues to issue virtual memory transactions, transaction Mux 420(0) routes the virtual memory transactions to uTLB 430 for processing.

In alternative embodiments, virtual memory transactions may be routed to any physical memory accessible to PPU 202 instead of PPU memory 204 . For example, virtual memory transactions may be routed to shared pages contained in system memory 104 .

Figure 5 is a flowchart of method steps for managing memory transactions issued by a Streaming Multiprocessor (SM), according to one embodiment of the present invention. Although these method steps are described herein in conjunction with FIGS. 1-4 , those skilled in the art will appreciate that any system configured to perform these method steps, in any order, falls within the scope of the present invention.

As shown, method 500 begins at step 502, where replay unit 350(0) receives a virtual memory transaction from SM 310(0). In response to receiving a virtual memory transaction from SM 310 ( 0 ), transaction Mux 420 contained in replay unit 350 ( 0 ) routes the virtual memory transaction to uTLB 430 and queues the virtual memory transaction in in-flight buffer 440 . If at step 504 uTLB 430 successfully processed the virtual memory transaction, method 500 proceeds to step 506 . At step 506 , fault detector 450 included in replay unit 350 ( 0 ) routes the corresponding physical memory transaction to PPU memory 204 , and method 500 returns to step 502 . Replay unit 350(0) loops through steps 502-506, receiving and processing virtual memory transactions from SM 310(0), until uTLB 430 is unable to successfully process virtual memory transactions from SM 310(0).

At step 504 , if uTLB 430 did not successfully map the virtual memory transaction to a physical address, method 500 proceeds to step 508 . At step 508 , fault detector 450 included in replay unit 350 ( 0 ) sends a fault signal to CPU 102 , stalls SM 310 ( 0 ), and adds the faulted virtual memory transaction to replay buffer 460 . At step 510 , fault detector 450 processes any virtual memory transactions queued in in-flight buffer 440 . This type of memory transaction corresponds to the memory transaction issued by SM 310(0), and begins execution before the failed virtual memory transaction. If any of these virtual memory transactions from SM 310 ( 0 ) also fail, then failure detector 450 performs one or more write operations, storing the additional failed virtual memory transactions in replay buffer 460 .

At step 512, replay unit 350(0) waits for CPU 102 to signal via the replay signal that one or more faults have been resolved. Upon receipt of the replay signal, replay unit 350 ( 0 ) invalidates uTLB 430 and re-executes the virtual memory transaction stored in replay buffer 460 . If uTLB 430 successfully maps a virtual memory transaction contained in replay buffer 460 , replay unit 350 ( 0 ) routes the corresponding physical memory transaction to PPU memory 204 . However, if uTLB 430 is unable to map a particular virtual memory transaction contained in replay buffer 460 , fault detector 450 performs a write operation to requeue the virtual memory transaction in replay buffer 460 . If, at step 514 , replay unit 350 ( 0 ) determines that replay buffer 460 is not empty, then method 500 returns to step 512 . Replay unit 350(0) loops through steps 512-514, re-executing the virtual memory transactions contained in replay buffer 460 until replay unit 350(0) determines that replay buffer 460 is empty.

At step 514 , if replay unit 350 ( 0 ) determines that replay buffer 460 is empty, method 500 proceeds to step 516 . At step 516 , replay unit 350 ( 0 ) causes SM 310 ( 0 ) to continue issuing virtual memory transactions from SM 310 ( 0 ), and method 500 returns to step 502 . Replay unit 350(0) continues to loop through steps 502-516, receiving and processing virtual memory transactions from SM 310(0).

In summary, Parallel Processing Units (PPUs) implement fault-handling techniques that allow certain Streaming Multiprocessors (SMs) to continue executing threads while causing other SMs to temporarily stop executing threads. In operation, if a memory access failure attributable to a thread executing on a particular SM occurs, the replay unit corresponding to the SM stalls the particular SM while the computer system resolves the failure. Note that the replay unit stops the corresponding SM from generating further memory transactions until the faulting memory transaction is resolved. In addition, the replay unit queues any in-flight memory transactions issued by the corresponding SM prior to the failure in the replay buffer. Once the fault is resolved, the replay unit causes the memory transactions stored in the replay buffer to be re-executed. After successfully executing all memory transactions stored in the replay unit, the replay unit enables the corresponding SM to continue generating additional memory transactions.

Advantageously, stalling affected SMs while allowing unaffected SMs to continue executing in the presence of memory access faults reduces the execution penalty associated with memory access faults. Note that the instruction does not need to be canceled because unaffected SMs continue to execute and the failed memory transaction is stored and replayed. Also, because the computer systems collectively execute fault resolution procedures for failed memory transactions in-flight, overall system performance is improved compared to resolving each fault individually. In contrast, upon a memory access fault, a conventional PPU stalls all SMs included in the PPU and cancels all subsequent memory transactions generated by the SMs. SMs included in such PPUs do not continue to issue memory transactions until the memory access failure is resolved. Thus, performance degradation associated with memory access failures is reduced in PPUs implementing selective memory transaction and replay techniques compared to conventional PPUs.

While the foregoing is directed to embodiments of the invention, other and further embodiments of the invention may be devised without departing from the essential scope thereof. For example, aspects of the present invention can be implemented in hardware or software or a combination of hardware and software. One embodiment of the invention can be implemented as a program product for use with a computer system. The program(s) of the program product define functions of the embodiments (including the methods described herein) and can be contained on a variety of computer-readable storage media. Exemplary computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory devices within a computer, such as compact disc read-only memory (CD-ROM) disks that can be read by a CD-ROM drive , flash memory, read-only memory (ROM) chips, or any type of solid-state non-volatile semiconductor memory) on which to store persistent information; and (ii) writable storage media (for example, a floppy disk or any type of solid-state random-access semiconductor memory) on which information can be changed. Such computer-readable storage media, when carrying computer-readable instructions for the functions of the present invention, are embodiments of the present invention.

The invention has been described above with reference to specific embodiments. Those of ordinary skill in the art will appreciate, however, that various modifications and changes can be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. Accordingly, the foregoing description and drawings are to be regarded in an illustrative rather than a restrictive sense.

Accordingly, the scope of the invention is defined by the appended claims.

Claims (10)

1. A computer-implemented method for processing virtual memory transactions associated with a multithreaded processing unit, the method comprising: receiving a first virtual memory transaction from a first unit; attempting to perform the first virtual memory transaction; detecting a first page fault associated with the first virtual memory transaction; storing the first virtual memory transaction in a replay buffer; raising a stall condition that prohibits the first unit from initiating subsequent virtual memory transactions until the first page fault has been resolved; and The first virtual memory transaction and at least one other virtual memory transaction stored in the replay buffer are re-executed once the first page fault has been resolved. 2. The method of claim 1, further comprising determining that the replay buffer is empty, and enabling the first unit to generate subsequent virtual memory transactions. 3. The method of claim 1, further comprising receiving a second virtual memory transaction from a second unit while the first page fault is unresolved, and successfully executing the second virtual memory transaction. 4. The method of claim 1, further comprising: receiving a second virtual memory transaction from the first unit prior to detecting the first page fault; detecting a second page fault associated with the second virtual memory transaction; and The second virtual memory transaction is stored in the replay buffer. 5. The method of claim 1, further comprising invalidating a translation lookaside cache prior to re-executing the first virtual memory transaction. 6. The method of claim 1, wherein re-executing the first virtual memory transaction comprises: determining whether an entry corresponding to the first virtual memory transaction exists in a translation lookaside cache; and If the entry exists, complete the first virtual memory transaction, otherwise If the entry does not exist, the first virtual memory transaction is restored in the replay buffer. 7. The method of claim 1, wherein resolving the first page fault comprises: placing a memory page associated with the first virtual memory transaction within a first memory subsystem based on a global translation table; and A virtual mapping for the memory page is added to a translation lookaside cache. 8. The method of claim 7, wherein addressing the first page fault further comprises copying the memory page from the first memory subsystem to a second memory subsystem. 9. The method of claim 8, wherein the first memory subsystem comprises a memory coupled to a central processing unit, and the second memory subsystem comprises a memory coupled to the multi-threaded processing unit of memory. 10. A system configured to process virtual memory transactions, the system comprising: memory; and a multithreaded processor coupled to the memory and configured to: receiving a first virtual memory transaction from a first unit; attempting to perform the first virtual memory transaction; detecting a first page fault associated with the first virtual memory transaction; storing the first virtual memory transaction in a replay buffer; raising a stall condition that prohibits the first unit from initiating subsequent virtual memory transactions until the first page fault has been resolved; and The first virtual memory transaction and at least one other virtual memory transaction stored in the replay buffer are re-executed once the first page fault has been resolved.