Add magic function %drun to run code in debugger #3071

tkf · 2013-03-24T17:44:35Z

%drun is pretty much like %prun but instead it starts debugger rather than profiler.

This PR is made on top of #3066. I will rebase it after #3066 is merged if needed.

takluyver · 2013-03-24T18:49:15Z

Is there an argument for this besides being quicker to type? The name drun isn't particularly self-explanatory.

tkf · 2013-03-24T19:00:17Z

If there was not %pdb or %debug, I would use it but I couldn't. So I thought I'd make it consistent with %prun. Another option I had in mind was %with_debugger (or the one without underscore).

takluyver · 2013-03-24T20:14:13Z

To my mind, a slightly longer, more descriptive name is preferable. Especially with the notebook, we're trending towards better readability, rather than saving every keystroke. But I'm not a big user of %run -d, so I'll leave it up to others.

minrk · 2013-03-24T20:21:03Z

We've generally been moving away from cryptic abbreviated names (e.g. replacing %hist with %history). Like @takluyver, I almost never %run -d, so I don't think it earns a first-class shortcut (I would actually say the same for %prun, were it not for historical reasons).

tkf · 2013-03-24T20:22:03Z

In the notebook I think you can't use debugger as it does not support STDIN. I'd use it mostly in terminal. Though I don't have strong opinion and I am OK with long name.

Carreau · 2013-03-24T20:26:20Z

In the notebook I think you can't use debugger as it does not support STDIN.

We are waiting for @minrk PR:

tkf · 2013-03-24T20:35:02Z

@Carreau coooool!

So we want to rename it to %with_debugger or another name? An alternative way to add this functionality is to have %exec magic command which is like %run but works for statements. Then we can integrate commands such as %prun, %timeit, %time, %capture etc. into this. This is probably better because you can do thing like "capturing output while profiling".

minrk · 2013-03-24T20:40:33Z

That PR is going to be dependent on #3011 (it's my stdin branch, if you want to check it out).

takluyver · 2013-03-24T21:02:46Z

Ah, I'd missed that it worked on statements, I thought you were just proposing a shortcut for %run -d.

Perhaps %debug_run or %rundebug. We could even overload the %debug magic, as it doesn't take any options at present, but maybe that's confusing.

minrk · 2013-03-24T21:39:33Z

I was thinking the same thing about extending %debug, but I, too, worry about too much overload. The fact that %drun works on statements makes it very clear to me that it should not have run anywhere in the name, because run really means running files in IPython.

So if we are extending %debug, we would get things like:

# old behavior
%debug 
# debug a statement:
%debug my_statement()
# debug a cell:
%%debug
def foo():
    1/0
foo()

That doesn't seem too confusing to me, as it strictly adds functionality and doesn't change anything, right? It also seems fairly clear, as the sentence structure of debug statement actually describes what is going to happen. If anything, I think %debug is a better name for this new functionality than what it does currently.

tkf · 2013-03-24T22:12:38Z

I think overloading %debug is good idea.

The fact that %drun works on statements makes it very clear to me that it should not have run anywhere in the name

Then I think %prun should be renamed, because it does not act on python file. I took the idea of the name %drun from it.

takluyver · 2013-03-24T23:24:27Z

Unfortunately, %profile already exists and does something unrelated. I'm
not sure what a good alternative name would be. %profile_code?

minrk · 2013-03-24T23:26:15Z

It could be my own personal hangup about prun - I wouldn't touch it, and maybe others don't have this association of run == file.

tkf · 2013-03-24T23:45:59Z

I remember that I thought %prun is inconsistent with %run when I first saw it. How about %profile_exec or %exec -p? Or can't we just rename %profile to %ipy_profile or something like that? Current %profile seems not a function that is used in user's program.

takluyver · 2013-03-24T23:54:44Z

'exec' is a keyword (Py2) or a builtin function (Py3), so I think we should
avoid using it as a name. %profile_exec feels awkward to me because you're
using both words as verbs - I'd be likely to read it with profile as a
noun, and wonder how I can execute it. %profile_code or %profile_stmt seems
more clear.

If we were starting with a blank slate, I'd call the get-the-profile-name
function something like ipy_profile or profile_name. But I'm reluctant to
break existing expectations by renaming it now, especially as I think it's
been there since 0.11.

On 24 March 2013 23:46, Takafumi Arakaki notifications@github.com wrote:

I remember that I thought %prun is inconsistent with %run when I first
saw it. How about %profile_exec or %exec -p? Or can't we just rename
%profile to %ipy_profile or something like that? Current %profile seems
not a function that is used in user's program.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/3071#issuecomment-15372306
.

tkf · 2013-03-25T00:29:19Z

Ah, I forgot that you can omit %, so that's why you want to avoid exec.

IPython is about executing code so I think it does not make sense to have better and shorter name for kind of meta management magic function (%profile) than the one about executing code (%profile_code). Also, probably 1.0 is a good timing for changing this.

minrk · 2013-03-25T02:30:18Z

I don't have any attachment to the existing %profile magic - I doubt it has much use, and can be renamed to %show_profile or just removed.

minrk · 2013-03-29T03:53:03Z

I would like to get @fperez's opinion about the run nomenclature - it could be that what I said above is plain wrong, and this being identical to prun, but with d for debug, nothing actually needs to change.

fperez · 2013-04-26T04:04:03Z

I've been thinking about this quite a bit, and I think it's a good time to rationalize things out in the code execution magics department. Since running code is more or less the heart of ipython, we should do our best to have a clean and consistent api here.

I mostly concur with @minrk in the idea that run is for scripts, the others for code blocks; this wasn't really by explicit design but rather as an accident of history, but we might as well run with it as it lets us present a consistent message.

The rationale for what comes below: %run takes special flags for three things: timing, profiling and debugging. And we also several dedicated magics for those things. We should harmonize all this with cleaner names and an easy to understand mapping between %run and the dedicated functions, where the only difference is that %run applies to scripts/modules and the others are for local blocks (line/cell).

So here's what I think we should do:

Rename %profile to %active_profile or similar (current?). The word 'profile' is just too important to leave it to this slightly obscure use case.
Unify the proposed %drun with %debug as indicated above. Plain %debug would continue to work as today, but with arguments it would be used to debug simple statements or whole blocks (in cell magic mode).
Rename %prun to %profile, leaving %prun to just raise a UsageError about the renaming.
Unify %time and %timeit into a single function, named %time, that does effectively what %timeit does today. The simplistic behavior that %time has would be obtained with a -s flag or similar. There really shouldn't be two timing functions.
While we do this, the timing ones should grown an option to return the timing value(s) in seconds. I can't count the number of times I've needed this to automate timing comparisons.

In practice this plan really should be odne in separate PRs, one per topic (timing, profiling, debugging). I just wanted to lay out the plan here for feedback. Once we agree, this PR can focus on debugging and hopefully two more will come for profiling and timing.

How does this sound?

jstenar · 2013-04-26T05:46:38Z

How about using subcommands instead?

%run debug
%run timing
%run profile

fperez · 2013-04-27T02:16:57Z

@jstenar, you mean instead of %debug, %time, %profile? It seems to me that keeping them as separate commands for blocks is not a bad option, since they will be most likely used in the notebook/qtconsole, and that makes them short and readable. We're trying to make %run itself simpler, not more complex, and adding sub-command machinery into it seems to me like moving in the wrong direction...

Do you see any gains with your idea?

jstenar · 2013-04-29T19:30:30Z

I believe the discoverability would be better with subcommands. I mean %debug, %time, and %profile are just different ways of %run-ning code. To me it makes more sense since these commands just represent different ways of %running code.

I believe using subcommands can simplify the code. By switching to argparse and subcommands you can have a separate function for each subcommand instead of a giant function as it is now. However our argparse decorators would have to be updated to handle subcommands.

bfroehle · 2013-04-29T19:37:07Z

Discoverability could also be handled by improving the corresponding docstrings. For example %run could include something just like man pages do:

See Also
--------
    %debug, %profile, %time

I'm personally -1 to putting everything in %run. It's already way too complicated. Instead, what we should focus on is a nice class based design for these types of commands with several entry points (%run, %debug, %time, %profile) which can share common machinery.

fperez · 2013-04-29T23:22:33Z

I tend to agree with @bfroehle here, for two reasons:

run is already huge and complex, adding sub-command machinery into a magic is only going to make it worse.
It lets us keep the mental model of " run for scripts, explicit commands for code blocks". Which I think happens to also work visually quite well, because %run tends to be more used in the terminal/console/qtconsole, whereas the individual commands for multiline blocks are more of a notebook thing, and by being explicit words, they make for more readable notebooks in the long run.

@jstenar, does that seem reasonable?

jstenar · 2013-05-01T15:42:18Z

I do not have a strong opinion here.

But I think I'm a bit confused on the one hand it sounds like @bfroehle
wants to split out the debug, timing, and profile stuff from %run, and
@fperez seems to agree but it also sounds like point 2 of @fperez's
reasons is to have %run be used for scripts and %profile, %timing etc
for code blocks and not scripts. So what is the intention here should
%profile be used for codeblocks or scripts?

If we go for a design where %profile etc are separate commands that can
run scripts then it would make sense to remove that functionality from
%run in order to avoid duplication in code that easily will end up
behaving slightly differently over time.

Also I think @bfroehle's suggestion about the doc strings makes perfect
sense and will handle most of my concerns.

takluyver · 2013-05-01T16:31:07Z

To frame this a different way, there are at least three special kinds of execution (timed, profiled, debugged) that people might want for either a block of code in IPython, or code from an external file. What's the most intuitive way to handle that.

At present:

%run means run code from a file, and it takes flags -t, -d and -p for timing, debugging and profiling. Concise once you know it, but not especially obvious to newcomers.
There are separate %time[it], %prun (to become %profile?) and proposed %drun magics for using code inline. %prun and %drun are rather non-obvious.

Another possibility might be to introduce -f (file) flags to the execution magic commands, so you could do things like:

# Code inline
%profile do_something(x, y)

# Code from file
%profile -f script.py

takluyver · 2013-05-01T16:31:24Z

(Sorry, didn't mean to close)

tkf · 2013-05-03T21:33:32Z

How about %exe [options] CODE (or even %ex?) to execute (eval) code? %run supports specifying multiple execution "modes". For example, you can profile and measure timing simultaneously by %run -t -p FILE.py. This feature cannot be implemented if you choose "execution mode as a command" approach (i.e., %time, %profile and %debug). One negative point of %exe is that %exc CODE is equivalent to CODE so that you are always going to give at least one option. Probably command such as %debug can be an alias of %exe -d.

fperez · 2013-05-10T20:45:23Z

I'm -1 on %exe: I think %run is fairly close to common language usage and also has a long tradition in IPython, so it should stay as-is, but for timing/profiling/debugging of chunks of code, especially in the notebook, I much prefer clear and descriptive terms like %time or %debug to having notebook cells tagged with %exe -t, which is fairly cryptic.

Keep in mind that %run is mostly used at the terminal clients (including qt console), so it's more transient. But tagging cells for timing or profiling is more likely to be persistent, so I prefer to have self-explanatory terms used there.

ps - note that this PR needs a rebase. @tkf, if you can do that and the minor cleanups indicated above, I think we're close to merge point.

tkf · 2013-05-10T21:25:01Z

cells tagged with %exe -t, which is fairly cryptic

It is easy to add long options, like %exe --time. Also as I said, it is pretty straight forward to provide %time as an alias of %exe --time.

Keep in mind that %run is mostly used at the terminal clients

Well, I use %run a lot in notebook and I don't understand why it should not be used or discouraged in notebook. Yes, I know that explicit is good but in notebook you need to balance explicitness and conciseness. And I think IPython magic as the system to balance the two.

I am not focusing on naming issue. I agree that %time looks better than %exe -t. I am just pointing out that you loose ability to combine command if you go %debug/%time/... way.

fperez · 2013-05-10T22:16:03Z

You don't lose the ability to combine them, since it's perfectly reasonable to 'stack' magics (we do it all the time with %px):

%time
%profile

# ... code

I think that's fairly clear...

tkf · 2013-05-10T22:43:14Z

Current line magic does not work in that way, right?

%prun
os.path.join('a', 'b')

and

%prun os.path.join('a', 'b')

are not equivalent, I suppose.

%%time
%%prun
os.path.join('a', 'b')

does not work either.

If what you mean is to make one of the above work, that makes sense. I like the approach of stacking cell magics.

fperez · 2013-05-11T02:17:08Z

Yes, sorry, that's what I meant. They can all be updated to work as nested cell magics.

And btw, if you really want to have a single entry point for running blocks, rather than creating a new magic that needs to be taught, we can simply extend %run to also have cell magic behavior. It would then be:

%run [options] filename

as a line magic, and

%%run [options]
#...
# block of code
#...

as a cell one. That would reuse 100% of the %run implementation, extending it from files to blocks, without having to add a new command.

I actually think that's the cleanest solution: stackable individual magics for %time, %profile and %debug, and extending %run to work as a cell magic on blocks.

And all of these things should be reusing a few generic non-magic objects/functions, with the magics themselves being very thin syntax layers on top of these components.

fperez · 2013-05-11T03:20:29Z

Oh, and btw, this one needs a rebase...

tkf · 2013-05-11T14:33:59Z

I thought I'd wait until the discussion is settled, to avoid possibility that we decide to go other direction.

I also noticed that individual line magic has advantage over single line magic in other direction: You can avoid parsing options. %exe a -c is interpreted as code a and option -c rather than a - c. (This may be more important for %time and %timeit. If you want merge these line magics, you need to face this parsing problem.)

So, I can go ahead and implement %debug here, right? Then later we can merge it into %run when other magics like %profile and %time are ready.

tkf · 2013-05-26T19:19:10Z

I am not sure if the discussion is settled but I guessed this patch will be pulled by 90% so I just rebased it.

ellisonbg · 2013-06-15T22:04:51Z

@fperez can you look at this PR to see if all of the comment have been addressed?

fperez · 2013-07-12T22:26:10Z

@tkf, I'm sorry for the slow progress, my fault!!

I'd like to see a deeper cleanup/refactoring of our execution magics along the lines of the discussion above, but I think this particular PR is already progress in the right direction, and there's no point in holding it up further.

Thanks @tkf for the good work! If you're up for digging further into execution.py to continue post-1.0, by all means do so :)

Merging now.

Extend the `%debug` magic to also run a statement under the control of a debugger, if called with arguments. If called with no arguments after an error has occurred, it continues to work as before.

takluyver closed this May 1, 2013

takluyver reopened this May 1, 2013

tkf added 2 commits May 26, 2013 21:12

Add a mode to exec code in %debug magic

f042f3f

Document the new usage of %debug

32cb7ef

fperez merged commit e1ef88f into ipython:master Jul 12, 2013

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add magic function %drun to run code in debugger #3071

Add magic function %drun to run code in debugger #3071

Add magic function %drun to run code in debugger #3071

Add magic function %drun to run code in debugger #3071

Conversation