10000 urllib3 v2.4.0 on Python 3.13 doesn't work with EKS · Issue #2394 · kubernetes-client/python · GitHub
[go: up one dir, main page]
Skip to content

urllib3 v2.4.0 on Python 3.13 doesn't work with EKS #2394

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Tracked by #227
Tenzer opened this issue May 13, 2025 · 5 comments
Open
Tracked by #227

urllib3 v2.4.0 on Python 3.13 doesn't work with EKS #2394

Tenzer opened this issue May 13, 2025 · 5 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@Tenzer
Copy link
Tenzer commented May 13, 2025

What happened (please include outputs or screenshots):

The following exception is raised whenever calling the Kubernetes API of an EKS cluster:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='0123456789deadbeef01234567890000.gr7.us-east-1.eks.amazonaws.com', port=443): Max retries exceeded with url: /version/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1028)')))

Full stacktrace 
Traceback (most recent call last):
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
    ~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
    ~~~~~~~~~~~~^^
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connection.py", line 741, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
        sock=sock,
    ...<14 lines>...
        assert_fingerprint=self.assert_fingerprint,
    )
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connection.py", line 920, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
        sock=sock,
    ...<8 lines>...
        tls_in_tls=tls_in_tls,
    )
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/util/ssl_.py", line 480, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/util/ssl_.py", line 524, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/mk9waz7zbq8hxm6sxhwn88hhcwmgsa84-python3-3.13.3/lib/python3.13/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sock=sock,
        ^^^^^^^^^^
    ...<5 lines>...
        session=session
        ^^^^^^^^^^^^^^^
    )
    ^
  File "/nix/store/mk9waz7zbq8hxm6sxhwn88hhcwmgsa84-python3-3.13.3/lib/python3.13/ssl.py", line 1076, in _create
    self.do_handshake()
    ~~~~~~~~~~~~~~~~~^^
  File "/nix/store/mk9waz7zbq8hxm6sxhwn88hhcwmgsa84-python3-3.13.3/lib/python3.13/ssl.py", line 1372, in do_handshake
    self._sslobj.do_handshake()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1028)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
        conn,
    ...<10 lines>...
        **response_kw,
    )
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1028)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    client.get_code()
    ~~~~~~~~~~~~~~~^^
  File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api/version_api.py", line 61, in get_code
    return self.get_code_with_http_info(**kwargs)  # noqa: E501
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api/version_api.py", line 128, in get_code_with_http_info
    return self.api_client.call_api(
           ~~~~~~~~~~~~~~~~~~~~~~~~^
        '/version/', 'GET',
        ^^^^^^^^^^^^^^^^^^^
    ...<11 lines>...
        _request_timeout=local_var_params.get('_request_timeout'),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        collection_formats=collection_formats)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
                           path_params, query_params, header_params,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
                           _return_http_data_only, collection_formats,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           _preload_content, _request_timeout, _host)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
        method, url, query_params=query_params, headers=header_params,
        post_params=post_params, body=body,
        _preload_content=_preload_content,
        _request_timeout=_request_timeout)
  File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
           ~~~~~~~~~~~~~~~~~~~~^^^^^
                                query_params=query_params,
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^
                                _preload_content=_preload_content,
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                _request_timeout=_request_timeout,
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                headers=headers)
                                ^^^^^^^^^^^^^^^^
  File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/rest.py", line 244, in GET
    return self.request("GET", url,
           ~~~~~~~~~~~~^^^^^^^^^^^^
                        headers=headers,
                        ^^^^^^^^^^^^^^^^
                        _preload_content=_preload_content,
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        _request_timeout=_request_timeout,
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        query_params=query_params)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/rest.py", line 217, in request
    r = self.pool_manager.request(method, url,
                                  fields=query_params,
                                  preload_content=_preload_content,
                                  timeout=timeout,
                                  headers=headers)
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/_request_methods.py", line 135, in request
    return self.request_encode_url(
           ~~~~~~~~~~~~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<3 lines>...
        **urlopen_kw,
        ^^^^^^^^^^^^^
    )
    ^
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/_request_methods.py", line 182, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/poolmanager.py", line 443, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
           ~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<13 lines>...
        **response_kw,
        ^^^^^^^^^^^^^^
    )
    ^
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
           ~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<13 lines>...
        **response_kw,
        ^^^^^^^^^^^^^^
    )
    ^
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
           ~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<13 lines>...
        **response_kw,
        ^^^^^^^^^^^^^^
    )
    ^
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
        method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
    )
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='0123456789deadbeef01234567890000.gr7.us-east-1.eks.amazonaws.com', port=443): Max retries exceeded with url: /version/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1028)')))

What you expected to happen:

The exception shouldn't be raised and the call to the Kubernetes API should be made successfully.

How to reproduce it (as minimally and precisely as possible):

Use the latest version of this project with urllib3 v2.4.0 on Python 3.13.

import kubernetes
kubernetes.config.load_config()  # Should load a config for an EKS cluster
client = kubernetes.client.VersionApi()
client.get_code()

Anything else we need to know?:

This seems to be caused by the following change in urllib3 v2.4.0: issue, PR, which only takes effect on Python 3.13.

I've only experienced the issue with EKS, which must use self-signed certificates that aren't fully compatible with RFC 5280, notably because they don't provide an Authority Key Identifier.

I don't know if the same issue is the case of other Kubernetes providers.

Environment:

  • Kubernetes version (kubectl version): v1.32.3-eks-bcf3d70
  • OS (e.g., MacOS 10.13.6): macOS 15.4.1
  • Python version (python --version): 3.13.3
  • Python client version (pip list | grep kubernetes): 32.0.1
@Tenzer Tenzer added the kind/bug Categorizes issue or PR as related to a bug. label May 13, 2025
@Tenzer Tenzer mentioned this issue May 13, 2025
Open
@dblanchette
Copy link

We've run into this issue as well. AWS support told us this is only the case with EKS clusters created with version 1.16 and below. Creating a new cluster would fix the issue, but updating does not.

As a workaround:

client_configuration = None
if sys.version_info >= (3, 13):
    # https://docs.python.org/3/whatsnew/3.13.html#ssl
    client_configuration = Configuration()
    client_configuration.verify_ssl = False

api_client = ApiClient(configuration=client_configuration)
kubernetes.client.VersionApi(api_client)

Of course, it's insecure to completely disable SSL verification, but it does avoid the issue.

@rongxin-liu rongxin-liu mentioned this issue May 19, 2025
Open
5 tasks
paul-grundmann added a commit to paul-grundmann/k8s-gpu-cleaner-controller that referenced this issue May 19, 2025
@paul-grundmann
Downgraded to python 3.12 due to urllib certificate incompatibilities…
e0c389f 
… (s. kubernetes-client/python#2394)
@skaven81
Copy link

We ran into this as well, with an on-prem RKE 1.28 cluster.

The issue seems to be related to a change in behavior in the urllib3 library starting with 2.4.0: https://github.com/urllib3/urllib3/releases/tag/2.4.0

Added verify_flags option to create_urllib3_context with a default of VERIFY_X509_PARTIAL_CHAIN and VERIFY_X509_STRICT for Python 3.13+. (urllib3/urllib3#3571)

This seems to have changed the default behavior of TLS connections made with urllib3, specifically on Python 3.13+ Workarounds include dow 8000 ngrading to Python 3.12, or downgrading urllib3 to 2.3.0.

I suspect the fix for this in the kubernetes libary is to update calls to create_urllib3_context with verify_flags set to the previous default value.

@djmattyg007
Copy link

We're also running into this issue with an EKS cluster that was created back in 2019. We need to keep the updates flowing, and it seems like the easiest solution for now is to pin urllib3 to v2.3.0, but that's not a viable long-term solution. We would also appreciate a way to configure urllib3 to use the old behaviour, without disabling TLS verification entirely.

@yliaog
Copy link
Contributor
yliaog commented May 21, 2025

could you please file a PR for the fix? Thanks

@Tenzer
Copy link
Author
Tenzer commented May 21, 2025

What do you see an acceptable fix as being?

I guess you either can specify urllib3 should be a version <2.4.0, or perhaps expose an easy way to disable this new sticker check in urllib3?

@sekot sekot mentioned this issue May 22, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Tenzer @djmattyg007 @dblanchette @skaven81 @yliaog
0