8000 gh-132737: Support profiling modules that import __main___ by aneeshdurg · Pull Request #132738 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-132737: Support profiling modules that import __main___ #132738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Apr 24, 2025
Merged

gh-132737: Support profiling modules that import __main___ #132738

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
b4f3934
Support profiling modules that import __main___
aneeshdurg Apr 19, 2025
e068bea
📜🤖 Added by blurb_it.
blurb-it[bot] Apr 19, 2025
ca66a0e
replace __main__'s namespace instead of creating a new module
aneeshdurg Apr 20, 2025
1c621a0
Update Misc/NEWS.d/next/Core_and_Builtins/2025-04-19-18-07-34.gh-issu…
aneeshdurg Apr 20, 2025
923cb6e
quote __main__ and fix typo
aneeshdurg Apr 20, 2025
75a542e
Add regression test
aneeshdurg Apr 20, 2025
3665a7f
Only modify __main__ in CLI invocation
aneeshdurg Apr 20, 2025
e6d50bd
Avoid leaking cProfile scope into profiled code
aneeshdurg Apr 20, 2025
5607cb8
Remove unused import
aneeshdurg Apr 20, 2025
8d6f37b
Undo changes to module execution
aneeshdurg Apr 20, 2025
191445c
Revert quote change
aneeshdurg Apr 21, 2025
2fffb33
less hacky
aneeshdurg Apr 23, 2025
8f9b8a1
Remove unecessary inclusion of __builtins__ to globals
aneeshdurg Apr 23, 2025
226d78d
Requested test changes
aneeshdurg Apr 23, 2025
15d8674
Update Misc/NEWS.d/next/Core_and_Builtins/2025-04-19-18-07-34.gh-issu…
aneeshdurg Apr 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Only modify __main__ in CLI invocation
  • Loading branch information
@aneeshdurg
aneeshdurg committed Apr 20, 2025
commit 3665a7f3f676b6f9abe477a680c507d019c9f0bd
25 changes: 11 additions & 14 deletions Lib/cProfile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,25 +98,13 @@ def run(self, cmd):
return self.runctx(cmd, dict, dict)

def runctx(self, cmd, globals, locals):
# cmd has to run in __main__ namespace (or imports from __main__ will
# break). Clear __main__ and replace with the globals provided.
import __main__
# Save a reference to the current __main__ namespace so that we can
# restore it after cmd completes.
original_main = __main__.__dict__.copy()
__main__.__dict__.clear()
__main__.__dict__.update(globals)

self.enable()
try:
exec(cmd, __main__.__dict__, locals)
exec(cmd, globals, locals)
finally:
self.disable()
__main__.__dict__.clear()
__main__.__dict__.update(original_main)
return self


# This method is more useful to profile a single function call.
def runcall(self, func, /, *args, **kw):
self.enable()
Expand Down Expand Up @@ -192,10 +180,19 @@ def main():
'__cached__': None,
'__builtins__': __builtins__,
}
# cmd has to run in __main__ namespace (or imports from __main__ will
# break). Clear __main__ and replace with the globals provided.
import __main__
# Save a reference to the current __main__ namespace so that we can
# restore it after cmd completes.
original_main = __main__.__dict__.copy()
__main__.__dict__.update(globs)

try:
runctx(code, globs, None, options.outfile, options.sort)
runctx(code, __main__.__dict__, None, options.outfile, options.sort)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need to restore it? You are exiting the program anyway. Also this is not ideal either. This will include all the global variables to the script that is being profiled. We want print(locals()) to be basically the same with or without the profiler.

Copy link
Contributor Author
@aneeshdurg aneeshdurg Apr 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue is to juggle the global variables needed by the call to (and implementation of) runctx. It would be a lot easier if I could split cprofile into a module where __main__ only has the main function. Is that something I can do?
if not, it's still possible, just trickier/messier.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No you can't split it into its own module :( that's too much a change. I think the correct way to go is to make the full runctx path independent of any global variables.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found a much cleaner fix - just ensure that the "main" function isn't run in the __main__ namespace.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I'm not sure if this is too hacky. I did not find such pattern in other code. It looks like an acceptable solution but I don't know if there will be implications. I want to ask @vstinner about this as he probably knows a lot of interesting usages.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like this hack works as expected. But it's strange and surprising :-)

I tried but failed (test fails) to inject a new __main__ module in sys.modules and leave the cProfile module unchanged:

diff --git a/Lib/cProfile.py b/Lib/cProfile.py
index 6253755a9df..abc03fc61eb 100644
--- a/Lib/cProfile.py
+++ b/Lib/cProfile.py
@@ -166,6 +166,7 @@ def main():
                 'run_module': runpy.run_module,
                 'modname': args[0]
             }
+            modname = args[0]
         else:
             progname = args[0]
             sys.path.insert(0, os.path.dirname(progname))
@@ -181,14 +182,18 @@ def main():
                 '__cached__': None,
                 '__builtins__': __builtins__,
             }
+            modname = spec.name
+
         # cmd has to run in __main__ namespace (or imports from __main__ will
         # break). Clear __main__ and replace with the globals provided.
-        import __main__
-        __main__.__dict__.clear()
-        __main__.__dict__.update(globs)
+        import __main__ as cProfileMain
+        new_main = type(cProfileMain)(modname)
+        new_main.__dict__.clear()
+        new_main.__dict__.update(globs)
+        sys.modules['__main__'] = new_main
 
         try:
-            runctx(code, __main__.__dict__, None, options.outfile, options.sort)
+            runctx(code, new_main.__dict__, None, options.outfile, options.sort)
         except BrokenPipeError as exc:
             # Prevent "Exception ignored" during interpreter shutdown.
             sys.stdout = None

Copy link
Contributor Author
@aneeshdurg aneeshdurg Apr 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this hack what pdb already does implicitly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cpython/Lib/pdb.py

Line 2449 in de9deb7

dictionary of the module __main__ is used (see the explanation of

Ah, it's actually explicitly documented

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vstinner @gaogaotiantian I did a bit more poking around and I think I managed to get rid of any of the hacky-ness. cProfile's main remains untouched, and in the case where we execute a file, I create a new module, set it as main, and ensure that the globals dict is the dict of the new module. The tests pass.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the current solution looks much less hacky.

except BrokenPipeError as exc:
__main__.__dict__.clear()
__main__.__dict__.update(original_main)
# Prevent "Exception ignored" during interpreter shutdown.
sys.stdout = None
sys.exit(exc.errno)
Expand Down
Loading
0