8000 Executing command in pod fails when _request_timeout is passed in tuple format · Issue #2292 · kubernetes-client/python · GitHub
[go: up one dir, main page]

Skip to content

Executing command in pod fails when _request_timeout is passed in tuple format #2292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tusharrobin opened this issue Oct 8, 2024 · 13 comments
Assignees
Labels
help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@tusharrobin
Copy link
tusharrobin commented Oct 8, 2024

What happened (please include outputs or screenshots):

Executing command in pod fails if _request_timeout is passed as tuple.
The exact error is
Exception when calling CoreV1Api->connect_get_namespaced_pod_exec: (0) Reason: '<' not supported between instances of 'float' and 'tuple'

What you expected to happen:
Expected to run the command successfully.

How to reproduce it (as minimally and precisely as possible):

`from kubernetes import client, config
from kubernetes.stream import stream

def run_command_in_pod(namespace, pod_name, container_name, command, timeout):
config.load_kube_config()
c = client.Configuration()
c.debug = True
api_instance = client.CoreV1Api()

exec_command = [
    '/bin/sh',
    '-c',
    command
]

try:
    response = stream(api_instance.connect_get_namespaced_pod_exec,
                      pod_name,
                      namespace,
                      container=container_name,
                      command=exec_command,
                      stderr=True, stdin=False,
                      stdout=True, tty=False,
                      _request_timeout=timeout)
    print("Command output: ", response)
except client.exceptions.ApiException as e:
    print("Exception when calling CoreV1Api->connect_get_namespaced_pod_exec: %s\n" % e)

if name == 'main':
namespace = 'test'
pod_name = 'test'
container_name = 'test'
command = 'date'
timeout = (5, 10)

run_command_in_pod(namespace, pod_name, container_name, command, timeout)`

Anything else we need to know?:

Environment:

  • Kubernetes version (kubectl version): 1.30
  • OS (e.g., MacOS 10.13.6): Rocky 8.6
  • Python version (python --version) 3.9
  • Python client version (pip list | grep kubernetes) kubernetes-31.0.0
@tusharrobin tusharrobin added the kind/bug Categorizes issue or PR as related to a bug. label Oct 8, 2024
@roycaihw
Copy link
Member
roycaihw commented Oct 9, 2024

/assign @yliaog

@yliaog
Copy link
Contributor
yliaog commented Oct 9, 2024

timeout should be a float, not a tuple as the error suggests

Exception when calling CoreV1Api->connect_get_namespaced_pod_exec: (0) Reason: '<' not supported between instances of 'float' and 'tuple'

@tusharrobin
Copy link
Author
tusharrobin commented Oct 9, 2024

Hi @yliaog, Thanks for looking into this.

The documentation for the API mentions
:param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts.

Reference: https://raw.githubusercontent.com/kubernetes-client/python/refs/heads/master/kubernetes/client/api/core_v1_api.py

If there is another way to set connect_timeout, let me know. Thanks again !

@yliaog
Copy link
Contributor
yliaog commented Oct 9, 2024

@tusharrobin
Copy link
Author
tusharrobin commented Oct 9, 2024

In case if the pod is on a node which is network partitioned and if you execute command in a pod running on that node, it will take 30 seconds to timeout even if you set "_request_timeout" to be a lesser value.

If you use curl with "connect_timeout" to a smaller value, it does timeout after that duration so we need to set connect_timeout for the "connect_get_namespaced_pod_exec" api so that it can timeout during network partition.

The only option I see in kubernetes client is to set the _request_timeout to a tuple value which allows to set "connect_timeout" which is not working. Can you please check if there is an issue ?

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 8, 2025
@tusharrobin
Copy link
Author

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 8, 2025
@yliaog
Copy link
Contributor
yliaog commented Jan 9, 2025

while self.is_open() and time.time() - start < timeout:

I think the error was caused by the above line, where timeout was only allowed to be a float, not a tuple.

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 9, 2025
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 9, 2025
@MrEcco
Copy link
MrEcco commented May 16, 2025

Same problem as explained in #2292 (comment) and #2292 (comment). I haven't found any good enough workaround.

One extra detail: if you set _request_timeout=5, it anyway waits 30 seconds. This timeout is hardcoded (somewhere) for websocket negotiation.

@roycaihw
Copy link
Member

As mentioned in #2292 (comment), I think we need some parsing like here when handling the timeout here

/help

it will take 30 seconds to timeout even if you set "_request_timeout" to be a lesser value
if you set _request_timeout=5, it anyway waits 30 seconds

Does setting _request_timeout to a larger value (e.g. 120s) take effects? Or is it always hard-coded to 30s?

@k8s-ci-robot
Copy link
Contributor

@roycaihw:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

  • Why are we solving this issue?
  • To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
  • Does this issue have zero to low barrier of entry?
  • How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

As mentioned in #2292 (comment), I think we need some parsing like here when handling the timeout here

/help

it will take 30 seconds to timeout even if you set "_request_timeout" to be a lesser value
if you set _request_timeout=5, it anyway waits 30 seconds

Does setting _request_timeout to a larger value (e.g. 120s) take effects? Or is it always hard-coded to 30s?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label May 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Development

No branches or pull requests

6 participants
0