8000 [3.6] bpo-44394: Update libexpat copy to 2.4.1 (GH-26945) (GH-28042) … · python/cpython@910886a · GitHub
[go: up one dir, main page]
Skip to content

Commit 910886a

Browse files
ned-deilyned-deily
authored
[3.6] bpo-44394: Update libexpat copy to 2.4.1 (GH-26945) (GH-28042) (GH-28080)
Update the vendored copy of libexpat to 2.4.1 (from 2.2.8) to get the fix for the CVE-2013-0340 "Billion Laughs" vulnerability. This copy is most used on Windows and macOS. Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Łukasz Langa <lukasz@langa.pl>. (cherry picked from commit 3fc5d84)
1 parent 8934bb0 commit 910886a

23 files changed

+1389
-191
lines changed

‎Doc/library/xml.rst

Lines changed: 19 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -60,23 +60,27 @@ circumvent firewalls.
6060
The following table gives an overview of the known attacks and whether
6161
the various modules are vulnerable to them.
6262

63-
========================= ============== =============== ============== ============== ==============
64-
kind sax etree minidom pulldom xmlrpc
65-
========================= ============== =============== ============== ============== ==============
66-
billion laughs **Vulnerable** **Vulnerable** **Vulnerable** **Vulnerable** **Vulnerable**
67-
quadratic blowup **Vulnerable** **Vulnerable** **Vulnerable** **Vulnerable** **Vulnerable**
68-
external entity expansion Safe (4) Safe (1) Safe (2) Safe (4) Safe (3)
69-
`DTD`_ retrieval Safe (4) Safe Safe Safe (4) Safe
70-
decompression bomb Safe Safe Safe Safe **Vulnerable**
71-
========================= ============== =============== ============== ============== ==============
72-
73-
1. :mod:`xml.etree.ElementTree` doesn't expand external entities and raises a
63+
========================= ================== ================== ================== ================== ==================
64+
kind sax etree minidom pulldom xmlrpc
65+
========================= ================== ================== ================== ================== ==================
66+
billion laughs **Vulnerable** (1) **Vulnerable** (1) **Vulnerable** (1) **Vulnerable** (1) **Vulnerable** (1)
67+
quadratic blowup **Vulnerable** (1) **Vulnerable** (1) **Vulnerable** (1) **Vulnerable** (1) **Vulnerable** (1)
68+
external entity expansion Safe (5) Safe (2) Safe (3) Safe (5) Safe (4)
69+
`DTD`_ retrieval Safe (5) Safe Safe Safe (5) Safe
70+
decompression bomb Safe Safe Safe Safe **Vulnerable**
71+
========================= ================== ================== ================== ================== ==================
72+
73+
1. Expat 2.4.1 and newer is not vulnerable to the "billion laughs" and
74+
"quadratic blowup" vulnerabilities. Items still listed as vulnerable due to
75+
potential reliance on system-provided libraries. Check
76+
:data:`pyexpat.EXPAT_VERSION`.
77+
2. :mod:`xml.etree.ElementTree` doesn't expand external entities and raises a
7478
:exc:`ParserError` when an entity occurs.
75-
2. :mod:`xml.dom.minidom` doesn't expand external entities and simply returns
79+
3. :mod:`xml.dom.minidom` doesn't expand external entities and simply returns
7680
the unexpanded entity verbatim.
77-
3. :mod:`xmlrpclib` doesn't expand external entities and omits them.
78-
4. Since Python 3.8.0, external general entities are no longer processed by
79-
default since Python.
81+
4. :mod:`xmlrpclib` doesn't expand external entities and omits them.
82+
5. Since Python 3.6.7, external general entities are no longer processed by
83+
default.
8084

8185

8286
billion laughs / exponential entity expansion

‎Doc/whatsnew/3.6.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2181,7 +2181,7 @@ Changes in the Python API
21812181

21822182
* The functions in the :mod:`compileall` module now return booleans instead
21832183
of ``1`` or ``0`` to represent success or failure, respectively. Thanks to
2184-
booleans being a subclass of integers, this should only be an issue if you7
2184+
booleans being a subclass of integers, this should only be an issue if you
21852185
were doing identity checks for ``1`` or ``0``. See :issue:`25768`.
21862186

21872187
* Reading the :attr:`~urllib.parse.SplitResult.port` attribute of

‎Misc/NEWS.d/next/Security/2021-06-29-02-45-53.bpo-44394.A220N1.rst

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
Update the vendored copy of libexpat to 2.4.1 (from 2.2.8) to get the fix
2+
for the CVE-2013-0340 "Billion Laughs" vulnerability. This copy is most used
3+
on Windows and macOS.

‎Modules/expat/COPYING

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
Copyright (c) 1998-2000 Thai Open Source Software Center Ltd and Clark Cooper
2-
Copyright (c) 2001-2017 Expat maintainers
2+
Copyright (c) 2001-2019 Expat maintainers
33

44
Permission is hereby granted, free of charge, to any person obtaining
55
a copy of this software and associated documentation files (the

‎Modules/expat/ascii.h

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,11 @@
66
\___/_/\_\ .__/ \__,_|\__|
77
|_| XML parser
88
9-
Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
10-
Copyright (c) 2000-2017 Expat development team
9+
Copyright (c) 1999-2000 Thai Open Source Software Center Ltd
10+
Copyright (c) 2000 Clark Cooper <coopercc@users.sourceforge.net>
11+
Copyright (c) 2002 Fred L. Drake, Jr. <fdrake@users.sourceforge.net>
12+
Copyright (c) 2007 Karl Waclawek <karl@waclawek.net>
13+
Copyright (c) 2017 Sebastian Pipping <sebastian@pipping.org>
1114
Licensed under the MIT license:
1215
1316
Permission is hereby granted, free of charge, to any person obtaining

‎Modules/expat/asciitab.h

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,9 @@
77
|_| XML parser
88
99
Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
10-
Copyright (c) 2000-2017 Expat development team
10+
Copyright (c) 2000 Clark Cooper <coopercc@users.sourceforge.net>
11+
Copyright (c) 2002 Fred L. Drake, Jr. <fdrake@users.sourceforge.net>
12+
Copyright (c) 2017 Sebastian Pipping <sebastian@pipping.org>
1113
Licensed under the MIT license:
1214
1315
Permission is hereby granted, free of charge, to any person obtaining

‎Modules/expat/expat.h

Lines changed: 35 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,14 @@
77
|_| XML parser
88
99
Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
10-
Copyright (c) 2000-2017 Expat development team
10+
Copyright (c) 2000 Clark Cooper <coopercc@users.sourceforge.net>
11+
Copyright (c) 2000-2005 Fred L. Drake, Jr. <fdrake@users.sourceforge.net>
12+
Copyright (c) 2001-2002 Greg Stein <gstein@users.sourceforge.net>
13+
Copyright (c) 2002-2016 Karl Waclawek <karl@waclawek.net>
14+
Copyright (c) 2016-2021 Sebastian Pipping <sebastian@pipping.org>
15+
Copyright (c) 2016 Cristian Rodríguez <crrodriguez@opensuse.org>
16+
Copyright (c) 2016 Thomas Beutlich <tc@tbeu.de>
17+
Copyright (c) 2017 Rhodri James <rhodri@wildebeest.org.uk>
1118
Licensed under the MIT license:
1219
1320
Permission is hereby granted, free of charge, to any person obtaining
@@ -115,7 +122,11 @@ enum XML_Error {
115122
XML_ERROR_RESERVED_PREFIX_XMLNS,
116123
XML_ERROR_RESERVED_NAMESPACE_URI,
117124
/* Added in 2.2.1. */
118-
XML_ERROR_INVALID_ARGUMENT
125+
XML_ERROR_INVALID_ARGUMENT,
126+
/* Added in 2.3.0. */
127+
XML_ERROR_NO_BUFFER,
128+
/* Added in 2.4.0. */
129+
XML_ERROR_AMPLIFICATION_LIMIT_BREACH
119130
};
120131

121132
enum XML_Content_Type {
@@ -318,7 +329,7 @@ typedef void(XMLCALL *XML_EndDoctypeDeclHandler)(void *userData);
318329
319330
For internal entities (<!ENTITY foo "bar">), value will
320331
be non-NULL and systemId, publicID, and notationName will be NULL.
321-
The value string is NOT nul-terminated; the length is provided in
332+
The value string is NOT null-terminated; the length is provided in
322333
the value_length argument. Since it is legal to have zero-length
323334
values, do not use this argument to test for internal entities.
324335
@@ -513,7 +524,7 @@ typedef struct {
513524
Otherwise it must return XML_STATUS_ERROR.
514525
515526
If info does not describe a suitable encoding, then the parser will
516-
return an XML_UNKNOWN_ENCODING error.
527+
return an XML_ERROR_UNKNOWN_ENCODING error.
517528
*/
518529
typedef int(XMLCALL *XML_UnknownEncodingHandler)(void *encodingHandlerData,
519530
const XML_Char *name,
@@ -707,7 +718,7 @@ XML_GetBase(XML_Parser parser);
707718
/* Returns the number of the attribute/value pairs passed in last call
708719
to the XML_StartElementHandler that were specified in the start-tag
709720
rather than defaulted. Each attribute/value pair counts as 2; thus
710-
this correspondds to an index into the atts array passed to the
721+
this corresponds to an index into the atts array passed to the
711722
XML_StartElementHandler. Returns -1 if parser == NULL.
712723
*/
713724
XMLPARSEAPI(int)
@@ -716,7 +727,7 @@ XML_GetSpecifiedAttributeCount(XML_Parser parser);
716727
/* Returns the index of the ID attribute passed in the last call to
717728
XML_StartElementHandler, or -1 if there is no ID attribute or
718729
parser == NULL. Each attribute/value pair counts as 2; thus this
719-
correspondds to an index into the atts array passed to the
730+
corresponds to an index into the atts array passed to the
720731
XML_StartElementHandler.
721732
*/
722733
XMLPARSEAPI(int)
@@ -997,7 +1008,10 @@ enum XML_FeatureEnum {
9971008
XML_FEATURE_SIZEOF_XML_LCHAR,
9981009
XML_FEATURE_NS,
9991010
XML_FEATURE_LARGE_SIZE,
1000-
XML_FEATURE_ATTR_INFO
1011+
XML_FEATURE_ATTR_INFO,
1012+
/* Added in Expat 2.4.0. */
1013+
XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_MAXIMUM_AMPLIFICATION_DEFAULT,
1014+
XML_FEATURE_BILLION_LAUGHS_ATTACK_PROTECTION_ACTIVATION_THRESHOLD_DEFAULT
10011015
/* Additional features must be added to the end of this enum. */
10021016
};
10031017

@@ -1010,12 +1024,24 @@ typedef struct {
10101024
XMLPARSEAPI(const XML_Feature *)
10111025
XML_GetFeatureList(void);
10121026

1027+
#ifdef XML_DTD
1028+
/* Added in Expat 2.4.0. */
1029+
XMLPARSEAPI(XML_Bool)
1030+
XML_SetBillionLaughsAttackProtectionMaximumAmplification(
1031+
XML_Parser parser, float maximumAmplificationFactor);
1032+
1033+
/* Added in Expat 2.4.0. */
1034+
XMLPARSEAPI(XML_Bool)
1035+
XML_SetBillionLaughsAttackProtectionActivationThreshold(
1036+
XML_Parser parser, unsigned long long activationThresholdBytes);
1037+
#endif
1038+
10131039
/* Expat follows the semantic versioning convention.
10141040
See http://semver.org.
10151041
*/
10161042
#define XML_MAJOR_VERSION 2
1017-
#define XML_MINOR_VERSION 2
1018-
#define XML_MICRO_VERSION 8
1043+
#define XML_MINOR_VERSION 4
1044+
#define XML_MICRO_VERSION 1
10191045

10201046
#ifdef __cplusplus
10211047
}

‎Modules/expat/expat_external.h

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,14 @@
77
|_| XML parser
88
99
Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
10-
Copyright (c) 2000-2017 Expat development team
10+
Copyright (c) 2000 Clark Cooper <coopercc@users.sourceforge.net>
11+
Copyright (c) 2000-2004 Fred L. Drake, Jr. <fdrake@users.sourceforge.net>
12+
Copyright (c) 2001-2002 Greg Stein <gstein@users.sourceforge.net>
13+
Copyright (c) 2002-2006 Karl Waclawek <karl@waclawek.net>
14+
Copyright (c) 2016 Cristian Rodríguez <crrodriguez@opensuse.org>
15+
Copyright (c) 2016-2019 Sebastian Pipping <sebastian@pipping.org>
16+
Copyright (c) 2017 Rhodri James <rhodri@wildebeest.org.uk>
17+
Copyright (c) 2018 Yury Gribov <tetra2005@gmail.com>
1118
Licensed under the MIT license:
1219
1320
Permission is hereby granted, free of charge, to any person obtaining

‎Modules/expat/iasciitab.h

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,9 @@
77
|_| XML parser
88
99
Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
10-
Copyright (c) 2000-2017 Expat development team
10+
Copyright (c) 2000 Clark Cooper <coopercc@users.sourceforge.net>
11+
Copyright (c) 2002 Fred L. Drake, Jr. <fdrake@users.sourceforge.net>
12+
Copyright (c) 2017 Sebastian Pipping <sebastian@pipping.org>
1113
Licensed under the MIT license:
1214
1315
Permission is hereby granted, free of charge, to any person obtaining

‎Modules/expat/internal.h

Lines changed: 49 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,12 @@
2525
\___/_/\_\ .__/ \__,_|\__|
2626
|_| XML parser
2727
28-
Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
29-
Copyright (c) 2000-2017 Expat development team
28+
Copyright (c) 2002-2003 Fred L. Drake, Jr. <fdrake@users.sourceforge.net>
29+
Copyright (c) 2002-2006 Karl Waclawek <karl@waclawek.net>
30+
Copyright (c) 2003 Greg Stein <gstein@users.sourceforge.net>
31+
Copyright (c) 2016-2021 Sebastian Pipping <sebastian@pipping.org>
32+
Copyright (c) 2018 Yury Gribov <tetra2005@gmail.com>
33+
Copyright (c) 2019 David Loffredo <loffredo@steptools.com>
3034
Licensed under the MIT license:
3135
3236
Permission is hereby granted, free of charge, to any person obtaining
@@ -101,22 +105,58 @@
101105
# endif
102106
#endif
103107

108+
#include <limits.h> // ULONG_MAX
109+
110+
#if defined(_WIN32) && ! defined(__USE_MINGW_ANSI_STDIO)
111+
# define EXPAT_FMT_ULL(midpart) "%" midpart "I64u"
112+
# if defined(_WIN64) // Note: modifiers "td" and "zu" do not work for MinGW
113+
# define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "I64d"
114+
# define EXPAT_FMT_SIZE_T(midpart) "%" midpart "I64u"
115+
# else
116+
# define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "d"
117+
# define EXPAT_FMT_SIZE_T(midpart) "%" midpart "u"
118+
# endif
119+
#else
120+
# define EXPAT_FMT_ULL(midpart) "%" midpart "llu"
121+
# if ! defined(ULONG_MAX)
122+
# error Compiler did not define ULONG_MAX for us
123+
# elif ULONG_MAX == 18446744073709551615u // 2^64-1
124+
# define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "ld"
125+
# define EXPAT_FMT_SIZE_T(midpart) "%" midpart "lu"
126+
# else
127+
# define EXPAT_FMT_PTRDIFF_T(midpart) "%" midpart "d"
128+
# define EXPAT_FMT_SIZE_T(midpart) "%" midpart "u"
129+
# endif
130+
#endif
131+
104132
#ifndef UNUSED_P
105133
# define UNUSED_P(p) (void)p
106134
#endif
107135

136+
/* NOTE BEGIN If you ever patch these defaults to greater values
137+
for non-attack XML payload in your environment,
138+
please file a bug report with libexpat. Thank you!
139+
*/
140+
#define EXPAT_BILLION_LAUGHS_ATTACK_PROTECTION_MAXIMUM_AMPLIFICATION_DEFAULT \
141+
100.0f
142+
#define EXPAT_BILLION_LAUGHS_ATTACK_PROTECTION_ACTIVATION_THRESHOLD_DEFAULT \
143+
8388608 // 8 MiB, 2^23
144+
/* NOTE END */
145+
146+
#include "expat.h" // so we can use type XML_Parser below
147+
108148
#ifdef __cplusplus
109149
extern "C" {
110150
#endif
111151

112-
#ifdef XML_ENABLE_VISIBILITY
113-
# if XML_ENABLE_VISIBILITY
114-
__attribute__((visibility("default")))
115-
# endif
152+
void _INTERNAL_trim_to_complete_utf8_characters(const char *from,
153+
const char **fromLimRef);
154+
155+
#if defined(XML_DTD)
156+
unsigned long long testingAccountingGetCountBytesDirect(XML_Parser parser);
157+
unsigned long long testingAccountingGetCountBytesIndirect(XML_Parser parser);
158+
const char *unsignedCharToPrintable(unsigned char c);
116159
#endif
117-
void
118-
_INTERNAL_trim_to_complete_utf8_characters(const char *from,
119-
const char **fromLimRef);
120160

121161
#ifdef __cplusplus
122162
}

‎Modules/expat/latin1tab.h

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,9 @@
77
|_| XML parser
88
99
Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
10-
Copyright (c) 2000-2017 Expat development team
10+
Copyright (c) 2000 Clark Cooper <coopercc@users.sourceforge.net>
11+
Copyright (c) 2002 Fred L. Drake, Jr. <fdrake@users.sourceforge.net>
12+
Copyright (c) 2017 Sebastian Pipping <sebastian@pipping.org>
1113
Licensed under the MIT license:
1214
1315
Permission is hereby granted, free of charge, to any person obtaining

‎Modules/expat/nametab.h

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,8 @@
66
\___/_/\_\ .__/ \__,_|\__|
77
|_| XML parser
88
9-
Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
10-
Copyright (c) 2000-2017 Expat development team
9+
Copyright (c) 2000 Clark Cooper <coopercc@users.sourceforge.net>
10+
Copyright (c) 2017 Sebastian Pipping <sebastian@pipping.org>
1111
Licensed under the MIT license:
1212
1313
Permission is hereby granted, free of charge, to any person obtaining

‎Modules/expat/siphash.h

Lines changed: 4 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@
1111
* --------------------------------------------------------------------------
1212
* HISTORY:
1313
*
14+
* 2020-10-03 (Sebastian Pipping)
15+
* - Drop support for Visual Studio 9.0/2008 and earlier
16+
*
1417
* 2019-08-03 (Sebastian Pipping)
1518
* - Mark part of sip24_valid as to be excluded from clang-format
1619
* - Re-format code using clang-format 9
@@ -96,15 +99,7 @@
9699
#define SIPHASH_H
97100

98101
#include <stddef.h> /* size_t */
99-
100-
#if defined(_WIN32) && defined(_MSC_VER) && (_MSC_VER < 1600)
101-
/* For vs2003/7.1 up to vs2008/9.0; _MSC_VER 1600 is vs2010/10.0 */
102-
typedef unsigned __int8 uint8_t;
103-
typedef unsigned __int32 uint32_t;
104-
typedef unsigned __int64 uint64_t;
105-
#else
106-
# include <stdint.h> /* uint64_t uint32_t uint8_t */
107-
#endif
102+
#include <stdint.h> /* uint64_t uint32_t uint8_t */
108103

109104
/*
110105
* Workaround to not require a C++11 compiler for using ULL suffix

‎Modules/expat/utf8tab.h

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,9 @@
77
|_| XML parser
88
99
Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
10-
Copyright (c) 2000-2017 Expat development team
10+
Copyright (c) 2000 Clark Cooper <coopercc@users.sourceforge.net>
11+
Copyright (c) 2002 Fred L. Drake, Jr. <fdrake@users.sourceforge.net>
12+
Copyright (c) 2017 Sebastian Pipping <sebastian@pipping.org>
1113
Licensed under the MIT license:
1214
1315
Permission is hereby granted, free of charge, to any person obtaining

‎Modules/expat/winconfig.h

Lines changed: 4 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,10 @@
66
\___/_/\_\ .__/ \__,_|\__|
77
|_| XML parser
88
9-
Copyright (c) 1997-2000 Thai Open Source Software Center Ltd
10-
Copyright (c) 2000-2017 Expat development team
9+
Copyright (c) 2000 Clark Cooper <coopercc@users.sourceforge.net>
10+
Copyright (c) 2002 Greg Stein <gstein@users.sourceforge.net>
11+
Copyright (c) 2005 Karl Waclawek <karl@waclawek.net>
12+
Copyright (c) 2017-2021 Sebastian Pipping <sebastian@pipping.org>
1113
Licensed under the MIT license:
1214
1315
Permission is hereby granted, free of charge, to any person obtaining
@@ -40,17 +42,4 @@
4042
#include <memory.h>
4143
#include <string.h>
4244

43-
#if defined(HAVE_EXPAT_CONFIG_H) /* e.g. MinGW */
44-
# include <expat_config.h>
45-
#else /* !defined(HAVE_EXPAT_CONFIG_H) */
46-
47-
# define XML_NS 1
48-
# define XML_DTD 1
49-
# define XML_CONTEXT_BYTES 1024
50-
51-
/* we will assume all Windows platforms are little endian */
52-
# define BYTEORDER 1234
53-
54-
#endif /* !defined(HAVE_EXPAT_CONFIG_H) */
55-
5645
#endif /* ndef WINCONFIG_H */

0 commit comments

Comments
 (0)
0