Global variables undefined in interactive use of embedded ipython shell #62
Comments
|
[ LP comment 1 by: Fernando Perez, on 2010-04-25 23:36:38.673176+00:00 ] OK, I can confirm the problem (even on trunk), but it's hard. I am just for now making sure this bug is confirmed so we keep tracking it, but I'm not sure how to fix it. The issue is that in embedded shells, we try to update the global namespace to use the surrounding scope (that's the point of an embedded shell, to be able to see what's around you). But this then causes python to fail to resolve the ipython interactive namespace when nested things (like local functions) are defined. See the mainloop() method of the embedded shell for details. I need to think a lot more about how to fix this one, any ideas very much welcome. |
|
Over at launchpad... chairmanK wrote 20 hours ago: Me Too. In addition to functions, this bug also appears in generator expressions. |
|
The equivalent component in trunk seems to be IPython.frontend.terminal.InteractiveShellEmbed. But that's broken in other ways, and evidently hasn't been tested much. Does anyone know what its future is? |
|
Could this be the same issue as #136? |
|
This has been now fixed: |
|
Can someone explain a little more what went into the fix of this? I still encounter the issue, but only when I'm one layer removed via a method call, and only if I run from a script, not interactively. python 2.7.2 on OSX 10.6.8, ipython 0.11 and 0.12 both exhibit similar behavior (using 0.12 for this comment) This is a problem with our project which features (prominently) an embedded IPython shell. testembed.py from IPython import embed
def hi():
embed()
if __name__ == '__main__':
#embed()
hi()Run this on the command line with However, comment out the call to After poking around, I think it has to do with the Thoughts? |
|
This is somewhat complicated, but I believe that's a restriction in Python itself. In the failing case you show, This is probably because nested scopes were added to Python relatively late (they were a future import in 2.1, and always on in 2.2). |
|
Ok, I think I understand. Looks like we can't really do anything about this then, other than dumping what I would write interactively to a file and then reading that file back in. Probably too complex for what we want to be able to do interactively. Thanks, btw. |
|
Sorry to keep harping on here, but it just makes interactive sessions feel very clunky: Regular python: Regular IPython: Embedded IPython: I guess there isn't much that can be done, but I don't get why we can create a dynamic closure in regular python/ipython but not the embedded version. |
|
By the oddities of Python scoping, in standard Python/IPython, we're not actually creating a closure. In contrast, when you make a closure, Python attaches references to each variable over which it has closed, as determined by the compiler - but that only works when the inner and outer functions are compiled at the same time. It looks like this: This is possibly done to save memory - if the closure carried a reference to the local scope where it was defined, none of the variables from that scope could be freed while the closure was live. |
|
I'm using Python 2.7.2 and IPython 0.12.1 on OSX 10.7.3 and having this issue still. When I run |
|
There's not a lot we can directly do about that, but we should probably encourage third parties away from |
|
@takluyver Do you mean that better to use ipython directly in this case? |
|
It's possible for Django to start IPython in a way that won't cause this problem, but that's not the way we currently make easy. The problem occurs when IPython starts with separate local and global namespaces. There's no reason that Django requires separate local and global namespaces, but that's what calling For reference, here's the code in Django: |
|
@takluyver That makes sense, thank! I'm opening a Django ticket for this. |
|
@takluyver, now that we've merged embed_kernel so all the main pieces are in place, do you want to tackle a bit a cleanup of this to make slightly more fine-tuned uses easier? |
|
I'll have a look at what interface might make most sense. |
|
I am still having trouble with this issue. I've tried narrowing down exactly what's causing it, and the best I can determine is that it only happens on Ubuntu 12.04. I've tried all of the latest release versions on other servers, and it works fine. But any time I define a function in ipython or use %cpaste to paste in a function from another file, the inside of that function has no access to the global scope. It makes it basically impossible to do anything useful in terms of writing functions on the fly. |
|
I am still having this problem with IPython 0.13 when it is called from other tools (e.g. Django's debugsqlshell command). It's very frustrating that something as basic as defining a function is totally broken. |
|
I think embed() is the wrong interface for those to be using. embed() is |
|
Not only ubuntu. debian wheezy also shows that. |
|
FYI, the Django ticket @liokm created above is https://code.djangoproject.com/ticket/18204, which now points to https://code.djangoproject.com/ticket/17078, which looks to have been fixed in trunk. It should land in 1.5. |
|
I m having the same issue on Ubuntu with Ipython 0.13.2 |
|
@bkvirendra that is fixed in django 1.6 |
|
But 1.6 is not even stable yet! |
Software are not always stable, and between releases there might still be bugs. But there is nothing that should be fixed in IPython. Even if we do something here, the fix would not be in IPython stable before it is released either... |
|
will this ever get fixed? |
|
AFAIK, my comments from a few years ago stand:
|
|
I don't understand, and it makes me angry. |
|
sorry, I was frustrated that things which would work if you typed them into an empty .py file didn't work here but re-reading the history, it seems this is an issue with Django's embarrassingly I'm working a lot in obsolete version of Django (1.4) according to earlier comment, newer versions of Django shell use ipython differently and may not have the problem? eg https://code.djangoproject.com/ticket/17078 apologies for the misunderstanding |
|
Yup, I think it was fixed for Django 1.6. If you really can't upgrade, you might want to apply the fix manually. It looks like this was it: django/django@3570ff7 |
|
I found a workaround today that is posted at #10695. A simpler treatment for the case where the IPython is not embedded inside a function is shown at this thread. I am not an expert so please help to check the validity. |
Embedding IPython results in issues where import statements and other
globals changes in the shell environment don't work as expected.[1]
As an example, the following commands result in a NameError:
In [1]: import time
In [2]: def foo():
...: print(time.time())
...:
In [3]: foo()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-3-c19b6d9633cf> in <module>
----> 1 foo()
<ipython-input-2-a038e7f91c93> in foo()
1 def foo():
----> 2 print(time.time())
3
NameError: name 'time' is not defined
By switching to use "start_ipython()" instead, we get an environment
more similar to what we get by running `ipython` directly.
[1] ipython/ipython#62
Embedding IPython results in issues where import statements and other
globals changes in the shell environment don't work as expected.[1]
As an example, the following commands result in a NameError:
In [1]: import time
In [2]: def foo():
...: print(time.time())
...:
In [3]: foo()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-3-c19b6d9633cf> in <module>
----> 1 foo()
<ipython-input-2-a038e7f91c93> in foo()
1 def foo():
----> 2 print(time.time())
3
NameError: name 'time' is not defined
By switching to use "start_ipython()" instead, we get an environment
more similar to what we get by running `ipython` directly.
[1] ipython/ipython#62
Embedding IPython results in issues where import statements and other
globals changes in the shell environment don't work as expected.[1]
As an example, the following commands result in a NameError:
In [1]: import time
In [2]: def foo():
...: print(time.time())
...:
In [3]: foo()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-3-c19b6d9633cf> in <module>
----> 1 foo()
<ipython-input-2-a038e7f91c93> in foo()
1 def foo():
----> 2 print(time.time())
3
NameError: name 'time' is not defined
By switching to use "start_ipython()" instead, we get an environment
more similar to what we get by running `ipython` directly.
[1] ipython/ipython#62
Embedding IPython results in issues where import statements and other
globals changes in the shell environment don't work as expected.[1]
As an example, the following commands result in a NameError:
In [1]: import time
In [2]: def foo():
...: print(time.time())
...:
In [3]: foo()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-3-c19b6d9633cf> in <module>
----> 1 foo()
<ipython-input-2-a038e7f91c93> in foo()
1 def foo():
----> 2 print(time.time())
3
NameError: name 'time' is not defined
By switching to use "start_ipython()" instead, we get an environment
more similar to what we get by running `ipython` directly.
[1] ipython/ipython#62
Summary: This fixes issues like ipython/ipython#62. Reviewed By: zzl0 Differential Revision: D44759311 fbshipit-source-id: 5ce94040cef09ac67962c147e86711971c545f91
|
I have a workaround posted at https://gist.github.com/limwz01/7e06cacf31d4ed83aa74134971030967 which is not as limited as the current approaches, but it has several caveats. Allowing closures over local variables is done by using a transformer added def _ipy_magic(_ipy_magic, x, y, z):
statement0
statement1
return (last_expression, _ipy_magic.update(locals()))[0]
_ipy_magic(locals(), x, y, z)
What this means is that you can just paste in code as if you are really inside the function you had embedded in, and it should work most of the time. This is useful to test small snippets of code as it is or in the event a program throws an exception, check if a modified piece of code works without re-running the whole program. Borrowing some of the above examples, these work: |
|
I'm wondering if there is any change in Python or IPython that could provide a first-class work around for this issue. It even confuses and annoys @3b1b: https://youtu.be/rbu7Zu5X1zI?si=3gcC1r2k3WrfZu-I&t=1734 |
|
My workaround at https://github.com/limwz01/ipython_utils is clunky but works.. and I've been using it for a long while now. 10000 |
|
Yeah, I'm in the process of watching @3b1b video, and I know this is an ipython limitation. I'd love a PR, I unfortunately dont have the time to dive into it. My guess is that it's the same issue that affects ipdb where list comprehension don't work. @limwz01, if you think your utils can be ported to be default in IPython, please send a PR. |
|
@Carreau I've packaged it as a separate pip installable package ipython_utils as I really need more people to test it in real world usage first. It does a lot of complicated stuff behind the scenes and right now it messes up the stack trace a little and I've had some minor problems with restoring the excepthook. You're welcome to try it out and give me some feedback |
|
@gaogaotiantian recently pushed a similar solution to pdb in cpython itself: python/cpython#111094 His solution is a little too simplistic and did not handle all the edge cases, so it may not be suitable here. However, it shows that such a solution is actually acceptable. |
Feel free to submit an issue about the edge cases that |
|
@gaogaotiantian I use the following to illustrate the problems: #!/usr/bin/env python3.13
g0 = 0
def main():
import pdb
pdb.Pdb().set_trace()
if __name__ == "__main__":
main()The following are the problems:
(it does return a value for
(Pdb) print((lambda:len(""" (Pdb) global g; g=1 |
|
I can't seem to edit my post to correct some typos and cross-reference the issue, but anyway the issue has been submitted here: python/cpython#126958 |
|
Use from IPython import start_ipython
start_ipython(user_ns={})or from IPython.terminal.embed import InteractiveShellEmbed
InteractiveShellEmbed(user_ns={})()or import IPython
IPython.embed(user_ns={})Result: In [1]: a = 1
In [2]: def f(x):
...: return a * x
...:
In [3]: f(2)
Out[3]: 2 |
|
This issue is about the problems of embedding a shell in a function scope,
something you totally missed in your examples, @ur001
Edit: okay that wasn't exactly the initial poster's problem, but in a few posts down it becomes obvious that embedding in a function is the problem that is remaining after the "This has been now fixed" post.
|
Original Launchpad bug 399627: https://bugs.launchpad.net/ipython/+bug/399627
Reported by: h-fangohr (Hans Fangohr).
The error can be reproduced as follows:
Within the just started ipython session, global variables are sometimes not visible. Two examples are:
Example 1:
Example 2:
There is no error if "b=1; (lambda :b)()" is put in a script, and this script is executed using ipython's "run" command.
There is no error if "b=1; (lambda :b)()" is executed interactively after having started ipython.
The only way the error seems comes up is if an embedded IPython shell is started from a Python prompt AND the commands are executed interactively at the prompt.
I find the same error when trying ipython-0.9.1 (with python2.4).
The bug was reported by Olivier Klein to the nmag team (http://nmag.soton.ac.uk).
The text was updated successfully, but these errors were encountered: