Variable Not Defined When Using Dictionary Comprehension Inside a Function Scope [Pdb] #100185

e-baumer · 2022-12-12T02:35:28Z

Bug report

I receive a NameError for a variable defined in a function scope when trying to execute a dictionary comprehension in pdb. The variable is part of the local scope as seen with locals()

For the following code:

def somefunc(fltrlist):
    d1 = {
        "A": [1, 2, 3],
        "B": [4, 5, 6],
        "C": [7, 8, 9]
    }

    import pdb;pdb.set_trace()
    d2 = {
        k: v for k, v in d1.items() if k in fltrlist
    }

klist = ["A", "B"]
somefunc(klist)

When I enter pdb and try to run the dictionary comprehension I get the following:

➜ python3 scope_test.py 
> scope_test.py(10)somefunc()
-> d2 = {
(Pdb) {k: v for k, v in d1.items() if k in fltrlist}
*** NameError: name 'fltrlist' is not defined
(Pdb)

The variable is part of the local scope,

(Pdb) locals()
{'d1': {'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]}, 'pdb': <module 'pdb' from '/usr/lib/python3.10/pdb.py'>, 'fltrlist': ['A', 'B']}
(Pdb) fltrlist
['A', 'B']
(Pdb)

If I try the same thing but not defined within a function it works.

fltrlist  = ["A", "B"]
d1 = {
    "A": [1, 2, 3],
    "B": [4, 5, 6],
    "C": [7, 8, 9]
}

import pdb;pdb.set_trace()
d2 = {
    k: v for k, v in d1.items() if k in fltrlist
}

When in pdb the dictionary comprehension works fine.

➜ python3 scope_test.py
> scope_test.py(11)<module>()
-> d2 = {
(Pdb) {k: v for k, v in d1.items() if k in fltrlist}
{'A': [1, 2, 3], 'B': [4, 5, 6]}
(Pdb)

Your environment

Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Pop!_OS 22.04 (jammy) - 6.0.6-76060006-generic

The text was updated successfully, but these errors were encountered:

sobolevn · 2022-12-12T06:37:19Z

This can be reproduced with just compile and exec (this is how they are used in pdb):

def somefunc(fltrlist):
    d1 = {"A": [1, 2, 3]}
    source = '{k: v for k, v in d1.items() if k in fltrlist}'
    code = compile(source + '\n', '<stdin>', 'single')
    exec(code, globals(), locals())

klist = ["A", "B"]
somefunc(klist)

Outputs:

Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/ex.py", line 8, in <module>
    somefunc(klist)
  File "/Users/sobolev/Desktop/cpython/ex.py", line 5, in somefunc
    exec(code, globals(), locals())
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <dictcomp>
NameError: name 'fltrlist' is not defined

sobolevn · 2022-12-12T06:40:10Z

Looks like this happens because LOAD_GLOBAL is used for loading fltrlist, the result of dis.dis(code):

Disassembly of <code object <dictcomp> at 0x10a812240, file "<stdin>", line 1>:
  1           0 RESUME                   0
              2 BUILD_MAP                0
              4 LOAD_FAST                0 (.0)
        >>    6 FOR_ITER                18 (to 46)
             10 UNPACK_SEQUENCE          2
             14 STORE_FAST               1 (k)
             16 STORE_FAST               2 (v)
             18 LOAD_FAST                1 (k)
             20 LOAD_GLOBAL              0 (fltrlist)
             32 CONTAINS_OP              0
             34 POP_JUMP_IF_TRUE         1 (to 38)
             36 JUMP_BACKWARD           16 (to 6)
        >>   38 LOAD_FAST                1 (k)
             40 LOAD_FAST                2 (v)
             42 MAP_ADD                  2
             44 JUMP_BACKWARD           20 (to 6)
        >>   46 END_FOR
             48 RETURN_VALUE

sobolevn · 2022-12-12T06:43:26Z

One more note: this happens on all versions of python, starting from 3.6 (I don't have any earlier versions installed). And even 2.7 🥼

wrongnull · 2023-04-08T05:42:16Z

@sobolevn I think it is impossible to fix without reimplementing this builtin function due to compile knows nothing about enclosing scope and as a consequence generates incorrect instructions

terryjreedy · 2023-04-08T16:07:26Z

https://docs.python.org/3/reference/expressions.html#displays-for-lists-sets-and-dictionaries explains:

The iterable expression in the leftmost for clause is evaluated directly in the enclosing scope and then passed as an argument to the implicitly nested scope. Subsequent for clauses and any filter condition in the leftmost for clause cannot be evaluated in the enclosing scope ...

Note that the global scope is always available.

@e-baumer It would have been better to have asked for an explanation under Help at https://discuss.python.org/. The docs should be consulted before reporting puzzling behavior as a bug.

carljm · 2023-04-10T16:14:28Z

I think the original report from @e-baumer here is valid, but is a duplicate of #65360. The key factor that makes this confusing in this particular case is that comprehensions (the dict comprehension, in this case) are compiled as nested functions.

e-baumer added the type-bug An unexpected behavior, bug, or error label Dec 12, 2022

sobolevn added the stdlib Python modules in the Lib dir label Dec 12, 2022

wrongnull mentioned this issue Apr 8, 2023

Unexpected behaviour of "compile" function #103375

Closed

terryjreedy closed this as completed Apr 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Variable Not Defined When Using Dictionary Comprehension Inside a Function Scope [Pdb] #100185

Variable Not Defined When Using Dictionary Comprehension Inside a Function Scope [Pdb] #100185

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Variable Not Defined When Using Dictionary Comprehension Inside a Function Scope [Pdb] #100185

Variable Not Defined When Using Dictionary Comprehension Inside a Function Scope [Pdb] #100185

Comments

Bug report

Your environment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!