8000 gh-102555 Increase HTML standard compliance for closing comment tags by Privat33r-dev · Pull Request #117406 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-102555 Increase HTML standard compliance for closing comment tags #117406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from
Closed

gh-102555 Increase HTML standard compliance for closing comment tags #117406

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c91c5ce
Increase HTML standard compliance for closing comment tags
Privat33r-dev Mar 31, 2024
4536d8b
📜🤖 Added by blurb_it.
blurb-it[bot] Mar 31, 2024
9147ff6
Add more edge cases and tests
Privat33r-dev Apr 6, 2024
eafb2d2
Add invalid HTML comment closing tags test cases
Privat33r-dev Apr 6, 2024
9e687bf
Add html closing comment tags test cases
Privat33r-dev Apr 6, 2024
caba267
Add EOF abrupted comment tag case handling and tests
Privat33r-dev Apr 6, 2024
23d35c6
Merge branch 'main' into patch-1
Privat33r-dev Feb 13, 2025
File filter

Filter by extension

Filter by extension
Conversations 8000
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions Lib/_markupbase.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

_declname_match = re.compile(r'[a-zA-Z][-_.a-zA-Z0-9]*\s*').match
_declstringlit_match = re.compile(r'(\'[^\']*\'|"[^"]*")\s*').match
_commentclose = re.compile(r'--\s*>')
_commentclose = re.compile(r'--!?>')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would leave the \s*, even though I should double check what the HTML5 specs say exactly.

Copy link
Contributor Author
@Privat33r-dev Privat33r-dev Apr 1, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would leave the \s*, even though I should double check what the HTML5 specs say exactly.

I provided the links to HTML5 specification earlier and "\s*" mentioned nowhere, moreover, my tests with latest versions of Firefox and Chrome has shown that it's in fact an incorrect behaviour and is not considered a closing tag by modern browsers. Thus I see no reason in keeping it (nor spec, nor common practice).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://html.spec.whatwg.org/#comment-end-state is the section of the specs I was looking for. It does indeed mention the ! but not the spaces, so updating the code accordingly sounds good to me.

Do you want to add tests to check these (-->, --!>, -- >, --x>, --->, etc.) cases?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://html.spec.whatwg.org/#comment-end-state is the section of the specs I was looking for. It does indeed mention the ! but not the spaces, so updating the code accordingly sounds good to me.

Do you want to add tests to check these (-->, --!>, -- >, --x>, --->, etc.) cases?

I am thinking about improving the solution to even include <!-->, unexpected EOF and similar other test cases (that were mentioned in a similar PR), but at the moment, unfortunately, I am lacking time to work on this PR. Hopefully, in the week (or at the weekend at worst) I can add the test cases and change a few other parts of the code to handle even wider variety of edge cases.

_markedsectionclose = re.compile(r']\s*]\s*>')

# An analysis of the MS-Word extensions is available at
Expand Down Expand Up @@ -81,7 +81,7 @@ def parse_declaration(self, i):
# A simple, practical version could look like: ((name|stringlit) S*) + '>'
n = len(rawdata)
if rawdata[j:j+2] == '--': #comment
# Locate --.*-- as the body of the comment
# Locate the body of the comment.
return self.parse_comment(i)
elif rawdata[j] == '[': #marked section
# Locate [statusWord [...arbitrary SGML...]] as the body of the marked section
Expand Down Expand Up @@ -161,13 +161,19 @@ def parse_marked_section(self, i, report=1):
self.unknown_decl(rawdata[i+3: j])
return match.end(0)

# Internal -- parse comment, return length or -1 if not terminated
def parse_comment(self, i, report=1):
# Internal -- parse comment
# if end is True, returns EOF location if no close tag is found, otherwise
# return length or -1 if not terminated
def parse_comment(self, i, report=1, end=False):
rawdata = self.rawdata
if rawdata[i:i+4] != '<!--':
raise AssertionError('unexpected call to parse_comment()')
match = _commentclose.search(rawdata, i+4)
match = _commentclose.search(rawdata, i+2)
if not match:
if end:
if report:
self.handle_comment(rawdata[i+4:])
return len(rawdata)
return -1
if report:
j = match.start(0)
Expand Down
2 changes: 1 addition & 1 deletion Lib/html/parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ def goahead(self, end):
elif startswith("</", i):
k = self.parse_endtag(i)
elif startswith("<!--", i):
k = self.parse_comment(i)
k = self.parse_comment(i, end=end)
elif startswith("<?", i):
k = self.parse_pi(i)
elif startswith("<!", i):
Expand Down
24 changes: 22 additions & 2 deletions Lib/test/test_htmlparser.py
View file
Open in desktop
8000
Original file line number Diff line number Diff line change
Expand Up @@ -324,14 +324,34 @@ def test_comments(self):
'<!---->'
'<!----I have many hyphens---->'
'<!-- I have a > in the middle -->'
'<!-- and I have -- in the middle! -->')
'<!-- and I have -- in the middle! -->'
'<!--->'
'<!-->'
'<!--<!--->'
'<!--And I am so-called incorrectly-closed-comment--!>'
'<!--!>'
'<!---!>'
'<!--I have invalid attempt to close (space) -- >-->'
'<!--Me too (invalid character) --x>-->'
'<!--Me too (invalid characters) --cheese>-->'
'<!--EOF comment')
expected = [('comment', " I'm a valid comment "),
('comment', 'me too!'),
('comment', '--'),
('comment', ''),
('comment', '--I have many hyphens--'),
('comment', ' I have a > in the middle '),
('comment', ' and I have -- in the middle! ')]
('comment', ' and I have -- in the middle! '),
('comment', ''),
('comment', ''),
('comment', '<!-'),
('comment', 'And I am so-called incorrectly-closed-comment'),
('comment', ''),
('comment', ''),
('comment', 'I have invalid attempt to close (space) -- >'),
('comment', 'Me too (invalid character) --x>'),
('comment', 'Me too (invalid characters) --cheese>'),
('comment', 'EOF comment')]
self._run_check(html, expected)

def test_condcoms(self):
Expand Down
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Security/2024-03-31-14-57-20.gh-issue-102555.2P8jGn.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Follow the `parsing recommendation <https://html.spec.whatwg.org/multipage/parsing.html#parse-error-incorrectly-closed-comment>`_ and `standard <https://html.spec.whatwg.org/#comments>`_ for closing comment tag in the :mod:`html.parser`. Increased compliance leads to predictable behavior, thus enhancing security.
Loading
0