diff options
| author | Warner Losh <imp@FreeBSD.org> | 2024-07-23 23:02:33 +0000 |
|---|---|---|
| committer | Warner Losh <imp@FreeBSD.org> | 2024-07-23 23:04:03 +0000 |
| commit | bb7f7d5b5201cfe569fce79b0f325bec2cf38ad2 (patch) | |
| tree | 7e64cd5bd5d6359aa1f62aa7bbac0a5d8f14bcea | |
| parent | aa41354349c16ea7220893010df78b47d67d0f74 (diff) | |
| download | src-bb7f7d5b5201cfe569fce79b0f325bec2cf38ad2.tar.gz src-bb7f7d5b5201cfe569fce79b0f325bec2cf38ad2.zip | |
nvme: Warn if there's system interrupt issues.
Issue a warning if we have system interrupt issues. If you get this
warning, then we submitted a request, it timed out without an interrupt
being posted, but when we polled the card's completion, we found
completion events. This indicates that we're missing interrupts, and to
date all the times I've helped people track issues like this down it has
been a system issue, not an NVMe driver isseue.
Sponsored by: Netflix
Reviewed by: gallatin
Differential Revision: https://reviews.freebsd.org/D46031
| -rw-r--r-- | share/man/man4/nvme.4 | 9 | ||||
| -rw-r--r-- | sys/dev/nvme/nvme_private.h | 1 | ||||
| -rw-r--r-- | sys/dev/nvme/nvme_qpair.c | 9 |
3 files changed, 17 insertions, 2 deletions
diff --git a/share/man/man4/nvme.4 b/share/man/man4/nvme.4 index 011ff483c839..dcd2ec86f5fa 100644 --- a/share/man/man4/nvme.4 +++ b/share/man/man4/nvme.4 @@ -239,6 +239,15 @@ detects that the AHCI device supports RST and when it is enabled. See .Xr ahci 4 for more details. +.Sh DIAGNOSTICS +.Bl -diag +.It "nvme%d: System interrupt issues?" +The driver found a timed-out transaction had a pending completion record, +indicating an interrupt had not been delivered. +The system is either not configuring interrupts properly, or the system drops +them under load. +This message will appear at most once per boot per controller. +.El .Sh SEE ALSO .Xr nda 4 , .Xr nvd 4 , diff --git a/sys/dev/nvme/nvme_private.h b/sys/dev/nvme/nvme_private.h index ff08f6581db5..05b5f3189eb2 100644 --- a/sys/dev/nvme/nvme_private.h +++ b/sys/dev/nvme/nvme_private.h @@ -303,6 +303,7 @@ struct nvme_controller { bool is_failed; bool is_dying; + bool isr_warned; STAILQ_HEAD(, nvme_request) fail_req; /* Host Memory Buffer */ diff --git a/sys/dev/nvme/nvme_qpair.c b/sys/dev/nvme/nvme_qpair.c index c917b34dbe43..0c3a36d4d76f 100644 --- a/sys/dev/nvme/nvme_qpair.c +++ b/sys/dev/nvme/nvme_qpair.c @@ -1145,9 +1145,14 @@ do_reset: /* * There's a stale transaction at the start of the queue whose * deadline has passed. Poll the competions as a last-ditch - * effort in case an interrupt has been missed. + * effort in case an interrupt has been missed. Warn the user if + * transactions were found of possible interrupt issues, but + * just once per controller. */ - _nvme_qpair_process_completions(qpair); + if (_nvme_qpair_process_completions(qpair) && !ctrlr->isr_warned) { + nvme_printf(ctrlr, "System interrupt issues?\n"); + ctrlr->isr_warned = true; + } /* * Now that we've run the ISR, re-rheck to see if there's any |