8000 Cull idle kernels · Issue #6893 · jupyterlab/jupyterlab · GitHub
[go: up one dir, main page]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cull idle kernels #6893

Open
adriendelsalle opened this issue Jul 25, 2019 · 9 comments
Open

Cull idle kernels #6893

adriendelsalle opened this issue Jul 25, 2019 · 9 comments
Assignees
@jasongrout
Labels
status:Needs Discussion
Milestone
Future

Comments

@adriendelsalle
Copy link
Contributor

Hi
You will find hereafter a short description of what I think to be a bug.
Thanks for your amazing job!

bug description
The classic notebook cull_connected option seems to be not well handled by jupyterlab. Even when a notebook if closed, the kernel won't be culled if the cull_connected option is set to False. If set to True, the behaviour is as expected.

steps to reproduce

  • setup a fresh env with the following packages

  • create a jupyter_notebook_config.py at ~/.jupyter/ with the options listed bellow
    -- c.MappingKernelManager.cull_connected = False
    -- c.MappingKernelManager.cull_idle_time = 20
    -- c.MappingKernelManager.cull_interval = 20

  • launch the classic notebook with jupyter notebook

  • create 2 new notebooks and close only one of the two

  • check the logs in your console, the kernel of the closed notebook will be culled but not the other one

  • reproduce with jupyterlab lauching jupyter lab

  • see that the kernel of the closed notebook is never culled

env

  • ubuntu 19.04
  • Browser = firefox 66.0.3
  • conda = 4.6.14
  • jupyter
    -- jupyterlab = 1.0.2
    -- jupyterlab_server = 1.0.0
    -- jupyter_core = 4.5.0
    -- jupyter_client = 5.3.1
    -- notebook = 6.0.0

Could you please have a look to this case?
Don't hesitate if I missed a point..!
Thank you !
@jasongrout
Copy link
Contributor

I didn't know about this option. However, here's my guess about what is happening. In the classic notebook, closing a notebook is actually closing the browser tab and all javascript that was keeping a websocket open, so there is no connection to the kernel, so the culling can happen. However, in JLab the connection to the kernel is maintained in global state, and since closing a notebook in jlab is not closing the browser tab, that global state stays around, hence the websocket connection stays around. To close a notebook's kernel connection, go to the running tab and hit "shutdown".

So I think that the model doesn't map very well to JupyterLab, which is a single-page application managing multiple kernel connections.

Here are a couple of options (using some jlab-specific technical terms):

  1. When all notebook views are closed, close down the notebook context's websocket connection to the kernel (i.e., it's KernelConnection). A major refactoring that didn't make it into 1.0 exposed the notion of a kernel being "connected" or "disconnected" - that would probably be useful here.
  2. Have some other way to achieve the same aim of culling kernels without involving the server option. For example, there have been experiments in shutting down a kernel on closing a notebook tab.

@adriendelsalle
Copy link
Contributor Author

Thanks for this quick response @jasongrout !

I experienced a similar behaviour with the shutdown_no_activity_time option that allows to shutdown a single-user server if no activity is monitored (for both notebook kernels and terminals).
The shutdown only occurs when all the browser tabs connected to jupyterlab are closed. I expect the culling of the notebook kernels also happens when tabs are closed.

I'm using these options through jupyterhub where it's pretty important to manage kernels and servers to spare resources..! Do you agree these options should stay available?

Since it's possible to switch between classic notebook and lab, I would say that the behaviour of the options have to be strictly similar. The 2nd option would introduce such a different behaviour. Are you OK with that @jasongrout ?
Any ideas @fcollonval ?

@jasongrout
Copy link
Contributor

Do you agree these options should stay available?

I'm just clarifying 8000 that the option is still available, just that JLab is maintaining the connection so the behavior isn't triggered.

@jasongrout jasongrout modified the milestones: 1.1, 1.2 Aug 2, 2019
@jasongrout
Copy link
Contributor

Setting as 1.1 so we can discuss what we'd like to do with JupyterLab keeping connections open after documents are closed (note this is not about shutting down kernels, just about keeping websocket connections live).

@adriendelsalle
Copy link
Contributor Author

Thanks!
I understand that this issue could be handled as a side effect of the websocket connections behaviour. Culling is a pretty important feature for spawing JLab with JHub, limiting the resources consumed.

I saw possible related issues : #5914, #6142
It seems that each time a notebook is opened in JLab, a new websocket/session is created and never destroyed at the notebook closing event (close to a memory leak), having as consequences:

  • the connected property is always True because of many sessions remaining opened
  • if you run a cell with long time computation, close the notebook and reopen it before it finishes
    -- you see at reopen that your kernel is connected, but not it is busy: missing * in the cell, the bottom status and the other one top-right of the notebook at connected
    -- you have to wait for the computation stop to have the idle information
    -- you won't get the output (even if a buffer exists, we don't reconnect the same websocket)
    -> Is that behaviour related ?

@adriendelsalle adriendelsalle mentioned this issue Aug 19, 2019
Open
@blink1073 blink1073 modified the milestones: 1.1, 1.2 Aug 27, 2019
@jasongrout jasongrout modified the milestones: 1.2, 2.0 Oct 11, 2019
@jasongrout jasongrout self-assigned this Dec 2, 2019
@jasongrout jasongrout modified the milestones: 2.0, Future Jan 2, 2020
@igorng
Copy link
igorng commented Apr 28, 2020

Pleaseany update on this issue?
This is really important to manage resources...

@mikepqr mikepqr mentioned this issue Nov 19, 2020
Open
@cdibble
Copy link
cdibble commented Jun 9, 2021

In case this info is useful reference... Culling as handled by the idle-culler project will happily cull JLab servers that do not maintain an active browser connection, it seems. (related- I've opened this issue because I cannot seem to configure idle-culler to skip named servers).

Nevertheless, the more fine-grained management allowed by the configurations mentioned here would be very useful- allowing users to close their web connection and allow JLab servers to continue to run processes and only be culled when those processes finish. So anyway, just wanted to mention the idle-culler for reference and add a +1 for this feature request.

@cdibble cdibble mentioned this issue Jun 9, 2021
Closed
@meeseeksmachine
Copy link
Contributor

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/cull-kill-idle-jupyterhub-notebooks/10858/2

81E2 @viniciusdc viniciusdc mentioned this issue Mar 1, 2022
Merged
15 tasks
@mimizone
Copy link

+1
I still have some rogue JupyterLab servers not being culled as expected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jasongrout jasongrout

Labels
status:Needs Discussion
Projects
None yet
Milestone
Future
Development

No branches or pull requests
7 participants
@mimizone @jasongrout @blink1073 @igorng @cdibble @adriendelsalle @meeseeksmachine
0