8000 Improve pipeline for native commands · Issue #559 · PowerShell/PowerShell · GitHub
[go: up one dir, main page]
Skip to content

Improve pipeline for native commands #559

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
vors opened this issue Feb 19, 2016 · 10 comments
Closed

Improve pipeline for native commands #559

vors opened this issue Feb 19, 2016 · 10 comments
Assignees
@vors
Labels
Issue-Enhancement the issue is more of a feature request than a bug Resolution-Fixed The issue is fixed. Usability (LEGACY) Helps filter issues that might be higher priority because they directly affect usability WG-Engine core PowerShell engine, interpreter, and runtime
Milestone
6.0.0-alpha.13

Comments

@vors
Copy link
Collaborator
vors commented Feb 19, 2016

Consider

ping -c 4 ya.ru | grep 64

Run it in PS and in bash. Compare UX.
In PS output of ping is collected, then it's passed down the pipe all together.
In bash you see output of grep instantly after ping send one line to the pipe.

As @BrucePay pointed out it never appeared to be a problem on windows due to the lack of pipeable native command-line tools.

As a minimum, we should sort out this buffering flow.
Desirable: improve pipe speed for certain cases (i.e. native command to native command).

Open question: can we use a native pipe to provide a compatible speed, when pipeline doesn't involve PS commands?
@vors vors added the Usability (LEGACY) Helps filter issues that might be higher priority because they directly affect usability label Feb 20, 2016
@andyleejordan andyleejordan added this to the GA milestone May 4, 2016
@andyleejordan andyleejordan modified the milestones: v0.5.0, GA May 11, 2016
@andyleejordan andyleejordan modified the milestones: v0.5.0, v0.6.0, Aug17 Jun 8, 2016
@vors vors mentioned this issue Jul 2, 2016
Closed
@vors vors added the WG-Engine core PowerShell engine, interpreter, and runtime label Jul 2, 2016
@vors vors modified the milestones: v0.7.0, Aug17 Jul 3, 2016
@vors vors self-assigned this Jul 3, 2016
@vors vors mentioned this issue Jul 4, 2016
Closed
@vors vors mentioned this issue Jul 12, 2016
Closed
4 tasks
@andyleejordan andyleejordan mentioned this issue Jul 15, 2016
Closed
11 tasks
@vors vors modified the milestones: 6.0.0-alpha.8, v0.7.0 Jul 20, 2016
@vors
Copy link
Collaborator Author
vors commented Jul 20, 2016

Moving to a next milestone, it took longer then I anticipated.

@joeyaiello
Copy link
Contributor

How are you planning on addressing this one? AFAIK, the reason this is happening is that the pipeline waits for everything to get loaded into memory before it takes input.

There's another scenario where people want to pipe large amounts of binary data from a native binary and it's putting all of that binary data (5+ GBs) into memory before passing them to the next native command in PowerShell.

8000
@vors
Copy link
Collaborator Author
vors commented Jul 23, 2016 •
edited
Loading

I'm putting that on hold.
We will do that post-Aug17.
Current work-in-progress is stored in https://github.com/vors/PowerShell/tree/native-pipe

@vors vors modified the milestones: Future, 6.0.0-alpha.8 Jul 23, 2016
@be5invis
Copy link

Related: #1908
Piping through EXEs or EXE to file (or vice versa) currently should not parse the piped content as string, instead of redirecting it directly as bytes.

@vors
Copy link
Collaborator Author
vors commented Aug 19, 2016

@be5invis absolutely!

Current plan is to address it by two separate changes:

  1. Move start of native executable from pipeline Complete() method (executes at the time of end block) to Prepare() method (executes at the time of begin block). This is prototyped in https://github.com/vors/PowerShell/tree/native-pipe
  2. Combine sequential native pipline elements and treat them as one pipeline element. Link them with IO redirection directly instead of powershell pipeline.

Together these two changes will cover simple scenarios:

  • native | native
  • ps | native

Unfortunately native | ps and more complicated scenarios, like ps | native | ps, will still be sub-optimal. I'm looking forward for propositions to address them.

@be5invis
Copy link

@vors I suggest adding a “as-bytes” operator that accepts raw (bytes) input and redirecting from Native to PS can use this scanerio:

native | as-bytes | your-parser | ps

@lzybkr lzybkr mentioned this issue Aug 20, 2016
Closed
@GeeLaw GeeLaw mentioned this issue Sep 5, 2016
Closed
@lzybkr lzybkr added the WG-Engine-Performance core PowerShell engine, interpreter, and runtime performance label Sep 6, 2016
@jhclark jhclark mentioned this issue Sep 6, 2016
Closed
@vors vors modified the milestones: 6.0.0-Alpha.10, Future, 6.0.0, 6.0.0-alpha.10 Sep 6, 2016
@vors
Copy link
Collaborator Author
vors commented Sep 12, 2016

I'm going to narrow scope if this issue just to the first part:

Move start of native executable from pipeline Complete() method

For the native | native part, we will use #1908

@vors vors removed the WG-Engine-Performance core PowerShell engine, interpreter, and runtime performance label Sep 12, 2016
@vors
Copy link
Collaborator Author
vors commented Sep 12, 2016

Doing this work, I'm constantly having problems because of #1486
Since this change will affect a lot of code paths, I'd like to extensively run "Feature" tests locally in different systems and make sure that I don't introduce regressions. I'm considering this work blocked on #1486 and moving it to the next milestone to fix our test harness first.

@vors vors modified the milestones: 6.0.0-alpha.11, 6.0.0-alpha.10 Sep 12, 2016
@vors vors modified the milestones: 6.0.0-alpha.12, 6.0.0-alpha.11 Oct 3, 2016
@vors vors mentioned this issue Oct 11, 2016
Merged
5 tasks
@joeyaiello joeyaiello added the Resolution-Fixed The issue is fixed. label Nov 29, 2016
@joeyaiello joeyaiello mentioned this issue Jan 25, 2017
Closed
@joaoe
Copy link
joaoe commented Nov 24, 2019 •
edited
Loading

Does the fix work for the use case with fd redirection ?

command 2>&1 | other

PowerShell also mangles output when redirecting file descriptors.

@huettenhain huettenhain mentioned this issue Jun 20, 2021
Closed
@iSazonov iSazonov mentioned this issue Oct 8, 2022
Merged
29 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vors vors

Labels
Issue-Enhancement the issue is more of a feature request than a bug Resolution-Fixed The issue is fixed. Usability (LEGACY) Helps filter issues that might be higher priority because they directly affect usability WG-Engine core PowerShell engine, interpreter, and runtime
Projects
None yet
Milestone
6.0.0-alpha.13
Development

No branches or pull requests
10 participants
@Jaykul @be5invis @joaoe @vors @joeyaiello @lzybkr @andyleejordan @PowerShellTeam @SteveL-MSFT @mirichmo
0