Cache derive proc macro expansion with incremental query #145354

Kobzol · 2025-08-13T13:09:12Z

This is a revival of #129102, originally implemented by @futile. Since it looks like they are not active currently, I'd like to push this work forward.

The first commit is squashed and rebased work from the original PR, with author attribution to futile. The rest of the commits are some additional comments that I created mostly for myself to understand what happens here. I also did some cleanups based on Vadim's review comments on the original PR, plus I refactored the TLS access a bit using scoped_tls.

The biggest issue, as usually, are tests... I tried using #[rustc_clean(..., loaded_from_disk = "derive_macro_expansion")], but the problem is that since this query cannot recover the original key from its hash, and thus its fingerprintstyle is FingerprintStyle::Opaque, this crashes when I try to use loaded_from_disk. Any suggestions from someone who actually understands the query system would be welcome 😅

To answer one review question from the original PR: the Hash implementation for TokenStream is indeed called, and it is needed since the stream forms a part of a query key. Not sure about the Encoder Hash implementation though.

The last commit is WIP to enable the functionality by default for rustc-perf, I'll do another perf. run here.

TODO: document the new unstable flag

r? @petrochenkov

rustbot · 2025-08-13T13:09:17Z

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

Kobzol · 2025-08-13T13:09:30Z

@bors try @rust-timer queue

Cache derive proc macro expansion with incremental query

compiler/rustc_ast/src/tokenstream.rs

compiler/rustc_middle/src/query/mod.rs

bjorn3 · 2025-08-13T13:24:28Z

compiler/rustc_expand/src/proc_macro.rs

+    /// Must be called while the `enter` function is active.
+    fn with<F, R>(f: F) -> R
+    where
+        F: for<'a, 'b> FnOnce(&'b mut ExtCtxt<'a>, DeriveClient) -> R,


The ExtCtxt contains a bunch of things that would need to be tracked by the query system for sound caching. And the rest could either be retrieved from the tcx or be created from scratch to avoid having to use a thread local to bypass the query system.

This was discussed a bit in #129102 (comment) and a few comments right above it.

rust-bors · 2025-08-13T13:43:11Z

💔 Test for 21e746d failed: CI. Failed jobs:

try - dist-x86_64-linux (web logs, extended logs)

Kobzol · 2025-08-13T14:56:39Z

@bors try

Cache derive proc macro expansion with incremental query

rust-bors · 2025-08-13T15:28:31Z

💔 Test for b712f62 failed: CI. Failed jobs:

try - dist-x86_64-linux (web logs, extended logs)

cjgillot · 2025-08-13T19:21:03Z

compiler/rustc_expand/src/proc_macro.rs

+                let macro_def_id = invoc_expn_data.macro_def_id.unwrap();
+                let proc_macro_crate_hash = tcx.crate_hash(macro_def_id.krate);


This c 8000 an be done from inside the query, to force a dependency on crate_hash, not as part of the key.

Sorry, my knowledge of the query infrastructure is quite limited 😅 How would that work? The key would be just LocalExpnId and &TokenStream, and then inside the query I would just call tcx.crate_hash and throw away the results? 🤔

Kobzol · 2025-08-15T11:01:20Z

I looked into the benchmark failure on diesel, seems like I hold queries wrong somehow :) Any suggestions on what could be the case? Maybe the derived Hash impl for TokenStream does not correspond to the manual Eq impl? 🤔

thread 'rustc' panicked at /projects/personal/rust/rust/compiler/rustc_query_system/src/query/plumbing.rs:191:23:
active query job entry
stack backtrace:
   0: __rustc::rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::option::expect_failed
   3: <rustc_query_system::query::plumbing::JobOwner<(rustc_span::hygiene::LocalExpnId, rustc_data_structures::svh::Svh, &rustc_ast::tokenstream::TokenStream), rustc_query_system::query::QueryStackDeferred>>::complete::<rustc_query_system::query::caches::DefaultCache<(rustc_span::hygiene::LocalExpnId, rustc_data_structures::svh::Svh, &rustc_ast::tokenstream::TokenStream), rustc_middle::query::erase::Erased<[u8; 8]>>>
   4: rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::DefaultCache<(rustc_span::hygiene::LocalExpnId, rustc_data_structures::svh::Svh, &rustc_ast::tokenstream::TokenStream), rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, true>
   5: <scoped_tls::ScopedKey<rustc_expand::proc_macro::QueryDeriveExpandCtx>>::set::<<rustc_expand::proc_macro::QueryDeriveExpandCtx>::enter<<rustc_expand::proc_macro::DeriveProcMacro as rustc_expand::base::MultiItemModifier>::expand::{closure#1}::{closure#0}, core::result::Result<rustc_ast::tokenstream::TokenStream, ()>>::{closure#0}, core::result::Result<rustc_ast::tokenstream::TokenStream, ()>>
   6: <rustc_expand::proc_macro::QueryDeriveExpandCtx>::enter::<<rustc_expand::proc_macro::DeriveProcMacro as rustc_expand::base::MultiItemModifier>::expand::{closure#1}::{closure#0}, core::result::Result<rustc_ast::tokenstream::TokenStream, ()>>
   7: <rustc_expand::proc_macro::DeriveProcMacro as rustc_expand::base::MultiItemModifier>::expand
   8: <rustc_expand::expand::MacroExpander>::expand_invoc
   9: <rustc_expand::expand::MacroExpander>::fully_expand_fragment
  10: <rustc_expand::expand::MacroExpander>::expand_crate
  11: <rustc_session::session::Session>::time::<rustc_ast::ast::Crate, rustc_interface::passes::configure_and_expand::{closure#1}>
  12: rustc_interface::passes::resolver_for_lowering_raw
      [... omitted 6 frames ...]
  13: <rustc_middle::ty::context::TyCtxt>::resolver_for_lowering
  14: <std::thread::local::LocalKey<core::cell::Cell<*const ()>>>::with::<rustc_middle::ty::context::tls::enter_context<<rustc_middle::ty::context::GlobalCtxt>::enter<rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2}::{closure#0}, core::option::Option<rustc_interface::queries::Linker>>::{closure#1}, core::option::Option<rustc_interface::queries::Linker>>::{closure#0}, core::option::Option<rustc_interface::queries::Linker>>
  15: <rustc_middle::ty::context::TyCtxt>::create_global_ctxt::<core::option::Option<rustc_interface::queries::Linker>, rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_co
8000
mpiler::{closure#0}::{closure#2}>::{closure#2}::{closure#0}>
  16: <rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2} as core::ops::function::FnOnce<(&rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2})>>::call_once::{shim:vtable#0}
  17: <alloc::boxed::Box<dyn for<'a> core::ops::function::FnOnce<(&'a rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &'a std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt<'a>>, &'a rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena<'a>>, &'a rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena<'a>>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}), Output = core::option::Option<rustc_interface::queries::Linker>>> as core::ops::function::FnOnce<(&rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2})>>::call_once
  18: rustc_interface::passes::create_and_enter_global_ctxt::<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>
  19: std::panic::catch_unwind::<core::panic::unwind_safe::AssertUnwindSafe<rustc_interface::interface::run_compiler<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}::{closure#0}>, ()>
  20: <scoped_tls::ScopedKey<rustc_span::SessionGlobals>>::set::<rustc_interface::util::run_in_thread_with_globals<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}::{closure#0}, ()>
  21: rustc_span::create_session_globals_then::<(), rustc_interface::util::run_in_thread_with_globals<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}::{closure#0}>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace

Kobzol · 2025-08-25T14:54:25Z

Ok I did a bit of debugging, and found a fun fact: TokenStream == TokenStream is (or at least can be) false. So that has to be fixed before we can use token streams as query keys.. :)

Kobzol · 2025-08-25T14:57:04Z

Ok I found it, it's this. @nnethercote Do you think it's possible to make TokenStreams comparable?

petrochenkov · 2025-08-25T17:42:02Z

Technically, removing that PartialEq for InvisibleOrigin is a language change.
But this PR can be unblocked without the lang team involvement by migrating the macro_rules matching logic specifically from PartialEq to a custom comparison function.

I'm not actually sure where else PartialEq for TokenStream is used, besides the matching and now also queries.
UPD: Nowhere.
UPD2: I think (Partial)Eq for TokenStream can also be derived now.

Kobzol reacted with thumbs up emoji

Kobzol · 2025-08-25T18:48:25Z

Could you please specify what do you mean by "macro rules matching logic"? I tried removing PartialEq from InvisibleOrigin, which required removing it from Delimiter and TokenKind, and that makes a bunch of places in code not compile, because they use == on these types (approx. 13 usages across 4 files). All of that should be migrated to a custom trait?

I also thought that another alternative would be to wrap TokenStream in a newtype and implement Eq for that newtype, which would delegate to a custom comparison trait for the query key logic (so basically an inverse of what you proposed).

Cache derive proc macro expansion with incremental query

rust-bors · 2025-09-02T20:58:43Z

☀️ Try build successful (CI)
Build commit: ece5ebe (ece5ebeef97f4a37667ea16d920620594ad59481, parent: a2c8b0b92c14b02f0b3f96a0d5296f1090dc286b)

…ochenkov Derive `PartialEq` for `InvisibleOrigin` For rust-lang#145354, we need `PartialEq` for `TokenStream` to "just work". However, due to the special comparison implementation that was used for `InvisibleOrigin`, this wasn't the case. So I derived `PartialEq` for `InvisibleOrigin`, and used the previous special comparison logic only on the single place where it was actually required. r? `@petrochenkov`

…ochenkov Derive `PartialEq` for `InvisibleOrigin` For rust-lang#145354, we need `PartialEq` for `TokenStream` to "just work". However, due to the special comparison implementation that was used for `InvisibleOrigin`, this wasn't the case. So I derived `PartialEq` for `InvisibleOrigin`, and used the previous special comparison logic only on the single place where it was actually required. r? ``@petrochenkov``

rust-timer · 2025-09-02T22:58:09Z

Finished benchmarking commit (ece5ebe): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.6%	[0.1%, 1.3%]	21
Regressions ❌ (secondary)	0.4%	[0.2%, 0.6%]	4
Improvements ✅ (primary)	-2.4%	[-7.8%, -0.1%]	16
Improvements ✅ (secondary)	-0.5%	[-1.0%, -0.0%]	14
All ❌✅ (primary)	-0.7%	[-7.8%, 1.3%]	37

Max RSS (memory usage)

Results (primary 0.9%, secondary 2.6%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	1.6%	[0.7%, 2.8%]	5
Regressions ❌ (secondary)	2.6%	[2.6%, 2.6%]	1
Improvements ✅ (primary)	-0.8%	[-1.1%, -0.6%]	2
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.9%	[-1.1%, 2.8%]	7

Cycles

Results (primary -4.1%, secondary -3.3%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.3%	[2.3%, 2.3%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-5.2%	[-9.8%, -2.2%]	6
Improvements ✅ (secondary)	-3.3%	[-3.7%, -2.8%]	2
All ❌✅ (primary)	-4.1%	[-9.8%, 2.3%]	7

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 465.026s -> 466.363s (0.29%)
Artifact size: 388.33 MiB -> 388.36 MiB (0.01%)

…ochenkov Derive `PartialEq` for `InvisibleOrigin` For rust-lang#145354, we need `PartialEq` for `TokenStream` to "just work". However, due to the special comparison implementation that was used for `InvisibleOrigin`, this wasn't the case. So I derived `PartialEq` for `InvisibleOrigin`, and used the previous special comparison logic only on the single place where it was actually required. r? ```@petrochenkov```

…ochenkov Derive `PartialEq` for `InvisibleOrigin` For rust-lang#145354, we need `PartialEq` for `TokenStream` to "just work". However, due to the special comparison implementation that was used for `InvisibleOrigin`, this wasn't the case. So I derived `PartialEq` for `InvisibleOrigin`, and used the previous special comparison logic only on the single place where it was actually required. r? ````@petrochenkov````

…ochenkov Derive `PartialEq` for `InvisibleOrigin` For rust-lang#145354, we need `PartialEq` for `TokenStream` to "just work". However, due to the special comparison implementation that was used for `InvisibleOrigin`, this wasn't the case. So I derived `PartialEq` for `InvisibleOrigin`, and used the previous special comparison logic only on the single place where it was actually required. r? `````@petrochenkov`````

…ochenkov Derive `PartialEq` for `InvisibleOrigin` For rust-lang#145354, we need `PartialEq` for `TokenStream` to "just work". However, due to the special comparison implementation that was used for `InvisibleOrigin`, this wasn't the case. So I derived `PartialEq` for `InvisibleOrigin`, and used the previous special comparison logic only on the single place where it was actually required. r? ``````@petrochenkov``````

…ochenkov Derive `PartialEq` for `InvisibleOrigin` For rust-lang#145354, we need `PartialEq` for `TokenStream` to "just work". However, due to the special comparison implementation that was used for `InvisibleOrigin`, this wasn't the case. So I derived `PartialEq` for `InvisibleOrigin`, and used the previous special comparison logic only on the single place where it was actually required. r? ```````@petrochenkov```````

…ochenkov Derive `PartialEq` for `InvisibleOrigin` For rust-lang#145354, we need `PartialEq` for `TokenStream` to "just work". However, due to the special comparison implementation that was used for `InvisibleOrigin`, this wasn't the case. So I derived `PartialEq` for `InvisibleOrigin`, and used the previous special comparison logic only on the single place where it was actually required. r? ````````@petrochenkov````````

…ochenkov Derive `PartialEq` for `InvisibleOrigin` For rust-lang#145354, we need `PartialEq` for `TokenStream` to "just work". However, due to the special comparison implementation that was used for `InvisibleOrigin`, this wasn't the case. So I derived `PartialEq` for `InvisibleOrigin`, and used the previous special comparison logic only on the single place where it was actually required. r? `````````@petrochenkov`````````

Rollup merge of #146090 - Kobzol:invisible-origin-eq, r=petrochenkov Derive `PartialEq` for `InvisibleOrigin` For #145354, we need `PartialEq` for `TokenStream` to "just work". However, due to the special comparison implementation that was used for `InvisibleOrigin`, this wasn't the case. So I derived `PartialEq` for `InvisibleOrigin`, and used the previous special comparison logic only on the single place where it was actually required. r? `````````@petrochenkov`````````

rustbot · 2025-09-04T06:36:22Z

This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

The job aarch64-gnu-llvm-19-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)


---- [incremental] tests/incremental/derive_macro_expansion/proc_macro_unchanged.rs stdout ----
------rustc stdout------------------------------

------rustc stderr------------------------------
invoked

thread 'rustc' (229652) panicked at compiler/rustc_incremental/src/persist/dirty_clean.rs:388:91:
called `Result::unwrap()` on an `Err` value: ()
stack backtrace:
   0: __rustc::rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: <rustc_incremental::persist::dirty_clean::DirtyCleanVisitor>::check_item
   4: <std::thread::local::LocalKey<core::cell::Cell<*const ()>>>::with::<rustc_middle::ty::context::tls::enter_context<rustc_incremental::persist::dirty_clean::check_dirty_clean_annotations::{closure#0}, ()>::{closure#0}, ()>
   5: rustc_incremental::persist::dirty_clean::check_dirty_clean_annotations
   6: <rustc_session::session::Session>::time::<(), rustc_incremental::persist::save::save_dep_graph::{closure#0}::{closure#1}>
   7: <std::thread::local::LocalKey<core::cell::Cell<*const ()>>>::with::<rustc_middle::ty::context::tls::enter_context<rustc_incremental::persist::save::save_dep_graph::{closure#0}, ()>::{closure#0}, ()>
   8: rustc_incremental::persist::save::save_dep_graph
   9: <rustc_incremental::provide::{closure#0} as core::ops::function::FnOnce<(rustc_middle::ty::context::TyCtxt,)>>::call_once
  10: <rustc_middle::ty::context::TyCtxt>::finish
  11: <std::thread::local::LocalKey<core::cell::Cell<*const ()>>>::with::<rustc_middle::ty::context::tls::enter_context<<rustc_middle::ty::context::GlobalCtxt>::enter<rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2}::{closure#0}, core::option::Option<rustc_interface::queries::Linker>>::{closure#1}, core::option::Option<rustc_interface::queries::Linker>>::{closure#0}, core::option::Option<rustc_interface::queries::Linker>>
  12: <rustc_middle::ty::context::TyCtxt>::create_global_ctxt::<core::option::Option<rustc_interface::queries::Linker>, rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2}::{closure#0}>
  13: <rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2} as core::ops::function::FnOnce<(&rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2})>>::call_once::{shim:vtable#0}
  14: <alloc::boxed::Box<dyn for<'a> core::ops::function::FnOnce<(&'a rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &'a std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt<'a>>, &'a rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena<'a>>, &'a rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena<'a>>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}), Output = core::option::Option<rustc_interface::queries::Linker>>> as core::ops::function::FnOnce<(&rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2})>>::call_once
  15: rustc_interface::passes::create_and_enter_global_ctxt::<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>
  16: <scoped_tls::ScopedKey<rustc_span::SessionGlobals>>::set::<rustc_interface::util::run_in_thread_with_globals<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}::{closure#0}, ()>
  17: rustc_span::create_session_globals_then::<(), rustc_interface::util::run_in_thread_with_globals<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}::{closure#0}>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: the compiler unexpectedly panicked. this is a bug.

note: using internal features is not supported and expected to cause internal compiler errors when used incorrectly

note: rustc 1.91.0-nightly (365a22be6 2025-09-04) running on aarch64-unknown-linux-gnu

note: compiler flags: -Z threads=1 -Z simulate-remapped-rust-src-base=/rustc/FAKE_PREFIX -Z translate-remapped-path-to-local-path=no -Z ignore-directory-in-diagnostics-source-blocks=/cargo -Z ignore-directory-in-diagnostics-source-blocks=/checkout/vendor -C incremental=[REDACTED] -Z incremental-verify-ich -Z ui-testing -Z deduplicate-diagnostics=no -C prefer-dynamic -C rpath -C debuginfo=0 -Z query-dep-graph -Z cache-derive-macros

query stack during panic:
end of query stack

------------------------------------------

error in revision `rpass2`: compilation failed!
status: exit status: 101
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/tests/incremental/derive_macro_expansion/proc_macro_unchanged.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/checkout/vendor" "--sysroot" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2" "--target=aarch64-unknown-linux-gnu" "--cfg" "rpass2" "--check-cfg" "cfg(test,FALSE,rpass1,rpass2)" "-C" "incremental=/checkout/obj/build/aarch64-unknown-linux-gnu/test/incremental/derive_macro_expansion/proc_macro_unchanged/proc_macro_unchanged.inc" "-Z" "incremental-verify-ich" "-O" "--error-format" "json" "--json" "future-incompat" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/incremental/derive_macro_expansion/proc_macro_unchanged/a" "-A" "internal_features" "-A" "unused_parens" "-A" "unused_braces" "-Crpath" "-Cdebuginfo=0" "-Lnative=/checkout/obj/build/aarch64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/incremental/derive_macro_expansion/proc_macro_unchanged/auxiliary" "-Zquery-dep-graph" "-Zcache-derive-macros"
stdout: none
--- stderr -------------------------------
invoked

thread 'rustc' (229652) panicked at compiler/rustc_incremental/src/persist/dirty_clean.rs:388:91:
called `Result::unwrap()` on an `Err` value: ()
stack backtrace:
   0: __rustc::rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: <rustc_incremental::persist::dirty_clean::DirtyCleanVisitor>::check_item
   4: <std::thread::local::LocalKey<core::cell::Cell<*const ()>>>::with::<rustc_middle::ty::context::tls::enter_context<rustc_incremental::persist::dirty_clean::check_dirty_clean_annotations::{closure#0}, ()>::{closure#0}, ()>
   5: rustc_incremental::persist::dirty_clean::check_dirty_clean_annotations
   6: <rustc_session::session::Session>::time::<(), rustc_incremental::persist::save::save_dep_graph::{closure#0}::{closure#1}>
   7: <std::thread::local::LocalKey<core::cell::Cell<*const ()>>>::with::<rustc_middle::ty::context::tls::enter_context<rustc_incremental::persist::save::save_dep_graph::{closure#0}, ()>::{closure#0}, ()>
   8: rustc_incremental::persist::save::save_dep_graph
   9: <rustc_incremental::provide::{closure#0} as core::ops::function::FnOnce<(rustc_middle::ty::context::TyCtxt,)>>::call_once
  10: <rustc_middle::ty::context::TyCtxt>::finish
  11: <std::thread::local::LocalKey<core::cell::Cell<*const ()>>>::with::<rustc_middle::ty::context::tls::enter_context<<rustc_middle::ty::context::GlobalCtxt>::enter<rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2}::{closure#0}, core::option::Option<rustc_interface::queries::Linker>>::{closure#1}, core::option::Option<rustc_interface::queries::Linker>>::{closure#0}, core::option::Option<rustc_interface::queries::Linker>>
  12: <rustc_middle::ty::context::TyCtxt>::create_global_ctxt::<core::option::Option<rustc_interface::queries::Linker>, rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2}::{closure#0}>
  13: <rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2} as core::ops::function::FnOnce<(&rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2})>>::call_once::{shim:vtable#0}
  14: <alloc::boxed::Box<dyn for<'a> core::ops::function::FnOnce<(&'a rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &'a std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt<'a>>, &'a rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena<'a>>, &'a rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena<'a>>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}), Output = core::option::Option<rustc_interface::queries::Linker>>> as core::ops::function::FnOnce<(&rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2})>>::call_once
  15: rustc_interface::passes::create_and_enter_global_ctxt::<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>
  16: <scoped_tls::ScopedKey<rustc_span::SessionGlobals>>::set::<rustc_interface::util::run_in_thread_with_globals<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}::{closure#0}, ()>
  17: rustc_span::create_session_globals_then::<(), rustc_interface::util::run_in_thread_with_globals<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}::{closure#0}>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: the compiler unexpectedly panicked. this is a bug.

note: using internal features is not supported and expected to cause internal compiler errors when used incorrectly

note: rustc 1.91.0-nightly (365a22be6 2025-09-04) running on aarch64-unknown-linux-gnu

note: compiler flags: -Z threads=1 -Z simulate-remapped-rust-src-base=/rustc/FAKE_PREFIX -Z translate-remapped-path-to-local-path=no -Z ignore-directory-in-diagnostics-source-blocks=/cargo -Z ignore-directory-in-diagnostics-source-blocks=/checkout/vendor -C incremental=[REDACTED] -Z incremental-verify-ich -Z ui-testing -Z deduplicate-diagnostics=no -C prefer-dynamic -C rpath -C debuginfo=0 -Z query-dep-graph -Z cache-derive-macros

query stack during panic:
end of query stack
------------------------------------------

---- [incremental] tests/incremental/derive_macro_expansion/proc_macro_unchanged.rs stdout end ----

failures:
    [incremental] tests/incremental/derive_macro_expansion/proc_macro_unchanged.rs

test result: FAILED. 170 passed; 1 failed; 4 ignored; 0 measured; 0 filtered out; finished in 18.01s

Kobzol · 2025-09-04T07:21:01Z

#146090 has been merged. I'm still unsure how to properly test this though (as per the PR description).

Derive `PartialEq` for `InvisibleOrigin` For rust-lang/rust#145354, we need `PartialEq` for `TokenStream` to "just work". However, due to the special comparison implementation that was used for `InvisibleOrigin`, this wasn't the case. So I derived `PartialEq` for `InvisibleOrigin`, and used the previous special comparison logic only on the single place where it was actually required. r? `````````@petrochenkov`````````

rustbot assigned petrochenkov Aug 13, 2025

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 13, 2025

rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Aug 13, 2025

This comment has been minimized.

Sign in to view

rust-bors bot added a commit that referenced this pull request Aug 13, 2025

Auto merge of #145354 - Kobzol:cache-proc-derive-macros, r=<try>

21e746d

Cache derive proc macro expansion with incremental query

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 13, 2025

petrochenkov mentioned this pull request Aug 13, 2025

Experimental: Add Derive Proc-Macro Caching #129102

Closed

3 tasks

bjorn3 reviewed Aug 13, 2025

View reviewed changes

compiler/rustc_ast/src/tokenstream.rs Outdated Show resolved Hide resolved

bjorn3 reviewed Aug 13, 2025

View reviewed changes

compiler/rustc_middle/src/query/mod.rs Outdated Show resolved Hide resolved

bjorn3 reviewed Aug 13, 2025

View reviewed changes

petrochenkov removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 13, 2025

cjgillot self-assigned this Aug 13, 2025

This comment has been minimized.

Sign in to view

rust-bors bot added a commit that referenced this pull request Aug 13, 2025

Auto merge of #145354 - Kobzol:cache-proc-derive-macros, r=<try>

b712f62

Cache derive proc macro expansion with incremental query

This comment has been minimized.

Sign in to view

cjgillot reviewed Aug 13, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

rust-bors bot added a commit that referenced this pull request Sep 2, 2025

Auto merge of #145354 - Kobzol:cache-proc-derive-macros, r=<try>

ece5ebe

Cache derive proc macro expansion with incremental query

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Sep 2, 2025

petrochenkov added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Sep 3, 2025

futile and others added 6 commits September 4, 2025 08:34

Implement incremental caching for derive macro expansions

1927b28

Add some comments and rename the option to -Zcache-derive-macros

64d9177

Refactor TLS access

cf9728d

Update test

f9f055b

Derive Hash for TokenStream

ea30f6d

WIP: temporarily enable cache-derive-macros by default for rustc-perf

356d426

Kobzol force-pushed the cache-proc-derive-macros branch from 8540970 to 356d426 Compare September 4, 2025 06:36

		let macro_def_id = invoc_expn_data.macro_def_id.unwrap();
		let proc_macro_crate_hash = tcx.crate_hash(macro_def_id.krate);

Cache derive proc macro expansion with incremental query #145354

Are you sure you want to change the base?

Cache derive proc macro expansion with incremental query #145354

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

Uh oh!

This comment has been minimized.

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!