10000 PushOutScanOutput crashes · Issue #5994 · Theano/Theano · GitHub
[go: up one dir, main page]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PushOutScanOutput crashes #5994

Open
rizar opened this issue May 31, 2017 · 1 comment
Open

PushOutScanOutput crashes #5994

rizar opened this issue May 31, 2017 · 1 comment
Labels
Optimization Scan

Comments

@rizar
Copy link
Contributor
rizar commented May 31, 2017

I see the following error message during optimization. I am using the latest Theano, device=cuda and optimizer=fast_run

2017-05-31 16:32:28,057: theano.gof.opt: ERROR: SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7faa1d96d190>
2017-05-31 16:32:28,057: theano.gof.opt: ERROR: Traceback:
2017-05-31 16:32:28,089: theano.gof.opt: ERROR: Traceback (most recent call last):
  File "/u/bahdanau/Dist/theano/theano/gof/opt.py", line 241, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/u/bahdanau/Dist/theano/theano/gof/opt.py", line 87, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/u/bahdanau/Dist/theano/theano/scan_module/scan_opt.py", line 685, in apply
    node = self.process_node(fgraph, node)
  File "/u/bahdanau/Dist/theano/theano/scan_module/scan_opt.py", line 745, in process_node
    node, args)
  File "/u/bahdanau/Dist/theano/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars
    add_as_nitsots)
  File "/u/bahdanau/Dist/theano/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs
    reason='scanOp_pushout_output')
  File "/u/bahdanau/Dist/theano/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
    chk = fgraph.replace_all_validate(replacements, reason)
  File "/u/bahdanau/Dist/theano/theano/gof/toolbox.py", line 543, in replace_all_validate
    fgraph.validate()
  File "/u/bahdanau/Dist/theano/theano/gof/toolbox.py", line 434, in validate_
    ret = fgraph.execute_callbacks('validate')
  File "/u/bahdanau/Dist/theano/theano/gof/fg.py", line 594, in execute_callbacks
    fn(self, *args, **kwargs)
  File "/u/bahdanau/Dist/theano/theano/gof/toolbox.py", line 600, in validate
    raise theano.gof.InconsistencyError("Trying to reintroduce a removed node")
InconsistencyError: Trying to reintroduce a removed node
@nouiz
Copy link
Member
nouiz commented Jun 5, 2017

To work around that (but it could slow down computation), can you try this Theano flag?

optimizer_excluding=scanOp_pushout_output

Do you have code that reproduce that? It will be hard to debug without that.

@lvapeab lvapeab mentioned this issue Jun 13, 2017
Closed
@Thrandis Thrandis mentioned this issue Sep 4, 2017
Open
@lvapeab lvapeab mentioned this issue Jul 28, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Optimization Scan
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lamblin @nouiz @rizar
0