Fix macro matching for NoDelim delimited sequences.
#96307
Conversation
PR rust-lang#95159 unintentionally changed the behaviour of declarative macro matching for `NoDelim` delimited sequences. - Before rust-lang#95159, delimiters were implicit in `mbe::Delimited`. When doing macro matching, normal delimiters were generated out of thin air as necessary, but `NoDelim` delimiters were not. This was done within `TokenTree::{get_tt,len}`. - After rust-lang#95159, the delimiters were explicit. There was an unintentional change whereby `NoDelim` delimiters were represented explicitly just like normal delimeters. - rust-lang#95555 introduced a new matcher representation (`MatcherLoc`) and the `NoDelim` delimiters were made explicit within it, just like `mbe::Delimited`. - rust-lang#95797 then changed `mbe::Delimited` back to having implicit delimiters, but because matching is now being done on `MatcherLoc`, the behavioural change persisted. The fix is simple: remove the explicit `NoDelim` delimiters in the `MatcherLoc` representation. This gives back the original behaviour. As for why this took so long to be found: it seems that `NoDelim` sequences are unusual. It took a macro that generates another macro (from the `yarte_lexer` crate, found via a crater run) to uncover this. Fixes rust-lang#96305.
|
The only problem here is there is no test case added, because the only test case I have spans two crates, and I don't know how to incorporate that into rustc's testing structure. |
|
You can use an |
|
This is unfortunate case, I'd say that #95159 fixed a bug. |
|
In general, nonterminal tokens from It would be better to report this case early, at macro definition site, when parsing the LHS of the macro definition. |
|
Given that the macro in #96305 correctly reports an error if used locally, I's say that
is the right course of action here. See |
Fair enough. I've added this information to #96305, and I will close this PR. |
PR #95159 unintentionally changed the behaviour of declarative macro
matching for
NoDelimdelimited sequences.TtParser#95159, delimiters were implicit inmbe::Delimited. Whendoing macro matching, normal delimiters were generated out of thin air
as necessary, but
NoDelimdelimiters were not. This was done withinTokenTree::{get_tt,len}.TtParser#95159, the delimiters were explicit. There was an unintentionalchange whereby
NoDelimdelimiters were represented explicitly justlike normal delimeters.
parse_tt#95555 introduced a new matcher representation (MatcherLoc) and theNoDelimdelimiters were made explicit within it, just likembe::Delimited.Delimited. #95797 then changedmbe::Delimitedback to having implicitdelimiters, but because matching is now being done on
MatcherLoc,the behavioural change persisted.
The fix is simple: remove the explicit
NoDelimdelimiters in theMatcherLocrepresentation. This gives back the original behaviour.As for why this took so long to be found: it seems that
NoDelimsequences are unusual. It took a macro that generates another macro
(from the
yarte_lexercrate, found via a crater run) to uncover this.Fixes #96305.
r? @petrochenkov