fix(lexer): Only allow horizontal whitespace in frontmatter #146106

epage · 2025-09-02T02:15:40Z

In writing up the reference for frontmatter, I realized that we probably
shouldn't be accepting Unicode Line Ending characters between the code
fence and infostring or trailing after the infostring or a code fence.

In digging into the unicode specification we use for Whitespace, it
divides it up into categories, so I'm deferring to what it says for
horizontal whitespace for what should be used within a line.

Note, I am leaving out support for Unicode Default Ignorable characters.
I figure that can be discussed outside of this change within the
reference and tracking issue.

Fixes #145971

Frontmatter tracking issue: #136889

In writing up the reference for frontmatter, I realized that we probably shouldn't be accepting Unicode Line Ending characters between the code fence and infostring or trailing after the infostring or a code fence. In digging into the unicode specification we use for Whitespace, it divides it up into categories, so I'm deferring to what it says for horizontal whitespace for what should be used within a line. Note, I am leaving out support for Unicde Default Ignorable characters. I figure that can be discussed outside of this change within the reference and tracking issue.

rustbot · 2025-09-02T02:15:44Z

r? @nnethercote

rustbot has assigned @nnethercote.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

nnethercote · 2025-09-02T04:43:28Z

r? @fee1-dead

tests/ui/frontmatter/frontmatter-contains-whitespace.rs

fee1-dead · 2025-09-03T01:45:10Z

@bors r+ rollup

bors · 2025-09-03T01:45:13Z

📌 Commit 6f0da97 has been approved by fee1-dead

It is now in the queue for this repository.

fix(lexer): Only allow horizontal whitespace in frontmatter In writing up the reference for frontmatter, I realized that we probably shouldn't be accepting Unicode Line Ending characters between the code fence and infostring or trailing after the infostring or a code fence. In digging into the unicode specification we use for Whitespace, it divides it up into categories, so I'm deferring to what it says for horizontal whitespace for what should be used within a line. Note, I am leaving out support for Unicode Default Ignorable characters. I figure that can be discussed outside of this change within the reference and tracking issue. Fixes rust-lang#145971 Frontmatter tracking issue: rust-lang#136889

Rollup of 8 pull requests Successful merges: - #139113 (unstable book: in a sanitizer example, check the code) - #145279 (Constify conversion traits (part 1)) - #145414 (unicode-table-generator refactors) - #145823 (editorconfig: don't use nonexistent syntax) - #145944 (std: Start supporting WASIp2 natively ) - #145961 (resolve: Avoid a regression from splitting prelude into two scopes) - #146032 (Explicity disable LSX feature for `loongarch64-unknown-none` target) - #146106 (fix(lexer): Only allow horizontal whitespace in frontmatter ) r? `@ghost` `@rustbot` modify labels: rollup

fix(lexer): Only allow horizontal whitespace in frontmatter In writing up the reference for frontmatter, I realized that we probably shouldn't be accepting Unicode Line Ending characters between the code fence and infostring or trailing after the infostring or a code fence. In digging into the unicode specification we use for Whitespace, it divides it up into categories, so I'm deferring to what it says for horizontal whitespace for what should be used within a line. Note, I am leaving out support for Unicode Default Ignorable characters. I figure that can be discussed outside of this change within the reference and tracking issue. Fixes rust-lang#145971 Frontmatter tracking issue: rust-lang#136889

Rollup of 9 pull requests Successful merges: - #145279 (Constify conversion traits (part 1)) - #145414 (unicode-table-generator refactors) - #145823 (editorconfig: don't use nonexistent syntax) - #145944 (std: Start supporting WASIp2 natively ) - #145961 (resolve: Avoid a regression from splitting prelude into two scopes) - #146032 (Explicity disable LSX feature for `loongarch64-unknown-none` target) - #146106 (fix(lexer): Only allow horizontal whitespace in frontmatter ) - #146112 (don't uppercase error messages) - #146154 (CI: rfl: move job forward to Linux v6.17-rc3 plus 2 commits) r? `@ghost` `@rustbot` modify labels: rollup

Rollup of 8 pull requests Successful merges: - #145279 (Constify conversion traits (part 1)) - #145414 (unicode-table-generator refactors) - #145823 (editorconfig: don't use nonexistent syntax) - #145944 (std: Start supporting WASIp2 natively ) - #145961 (resolve: Avoid a regression from splitting prelude into two scopes) - #146032 (Explicity disable LSX feature for `loongarch64-unknown-none` target) - #146106 (fix(lexer): Only allow horizontal whitespace in frontmatter ) - #146154 (CI: rfl: move job forward to Linux v6.17-rc3 plus 2 commits) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of #146106 - epage:whitespace, r=fee1-dead fix(lexer): Only allow horizontal whitespace in frontmatter In writing up the reference for frontmatter, I realized that we probably shouldn't be accepting Unicode Line Ending characters between the code fence and infostring or trailing after the infostring or a code fence. In digging into the unicode specification we use for Whitespace, it divides it up into categories, so I'm deferring to what it says for horizontal whitespace for what should be used within a line. Note, I am leaving out support for Unicode Default Ignorable characters. I figure that can be discussed outside of this change within the reference and tracking issue. Fixes #145971 Frontmatter tracking issue: #136889

This is to align with rust-lang/rust#146106. While no tests were affected by this, I definitely broke a lot of tests when making this change.

### What does this PR try to resolve? This is to align with rust-lang/rust#146106 and is part of #12207. ### How to test and review this PR? While no tests were affected by this, I definitely broke a lot of tests when making this change.

epage added 5 commits September 1, 2025 20:51

test(lexer): Ensure frontmatter w/ crlf works

b2962c3

test(lexer): Ensure frontmatter can contain unicode whitespace

159be09

test(lexer): Ensure tabs are accepted for horizontal whitespace

9304aa1

docs(lexer): Organize and document whitespace by Pattern_White_Space

428e413

epage added the F-frontmatter `#![feature(frontmatter)]` label Sep 2, 2025

rustbot assigned nnethercote Sep 2, 2025

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Sep 2, 2025

rustbot assigned fee1-dead and unassigned nnethercote Sep 2, 2025

bjorn3 reviewed Sep 2, 2025

View reviewed changes

tests/ui/frontmatter/frontmatter-contains-whitespace.rs Outdated Show resolved Hide resolved

epage force-pushed the whitespace branch from 3ff755f to 6f0da97 Compare September 2, 2025 13:33

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 3, 2025

tgross35 mentioned this pull request Sep 3, 2025

Rollup of 8 pull requests #146141

Closed

Zalathar mentioned this pull request Sep 3, 2025

Rollup of 9 pull requests #146157

Closed

Zalathar mentioned this pull request Sep 3, 2025

Rollup of 8 pull requests #146160

Merged

bors merged commit 56213a5 into rust-lang:master Sep 3, 2025
10 checks passed

rustbot added this to the 1.91.0 milestone Sep 3, 2025

epage deleted the whitespace branch September 4, 2025 13:38

This was referenced Sep 16, 2025

fix(frontatter): Only allow horizontal whitespace after fences rust-lang/cargo#15975

Merged

Stabilize cargo-script rust-lang/rust-project-goals#119

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(lexer): Only allow horizontal whitespace in frontmatter #146106

fix(lexer): Only allow horizontal whitespace in frontmatter #146106

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

fix(lexer): Only allow horizontal whitespace in frontmatter #146106

fix(lexer): Only allow horizontal whitespace in frontmatter #146106

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants