8000 Invoke-RestMethod/WebRequest: Support CTRL-C when reading data and connection hangs by stevenebutler · Pull Request #19330 · PowerShell/PowerShell · GitHub
[go: up one dir, main page]
Skip to content

Invoke-RestMethod/WebRequest: Support CTRL-C when reading data and connection hangs #19330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Apr 3, 2023
Merged

Invoke-RestMethod/WebRequest: Support CTRL-C when reading data and connection hangs #19330

merged 3 commits into from
Apr 3, 2023

Conversation

stevenebutler
Copy link
Contributor
@stevenebutler stevenebutler commented Mar 15, 2023 •
edited
Loading

APIs that should be cancelled are passed the cancellation token so that CTRL-C can cancel even when processing long or timed-out response streams.

Switched use of synchronous stream APIs to asynchronous APIs that use the cancellation token.

PR Summary

This PR closes #16145

The root cause of these issues was that once the response headers are read, the cancellation token was not always being passed to APIs that deal with reading the response stream.

In some scenarios, CTRL-C worked (where the cancellation token was being passed down), but this was not occurring in all scenarios and the use of blocking read APIs that do not take cancellation tokens prevented the early cancellation when CTRL-C was used.

PR Context

This PR helps because it allows all I/O in the web requests cmdlets to be cancelled if the user uses CTRL-C to terminate processing.

A previous iteration of this PR added a timeout, but was incorrect as the desired behaviour is to cancel if the connection is stalled for 5 minutes, not after a fixed timeout.

PR Checklist

@stevenebutler
Copy link
Contributor Author

@CarloToso and @iSazonov - I renamed the branch to better reflect what this is doing. This closed the PR #19315 I had open so I've opened this one instead for review. Changes are similar to what @CarloToso already reviewed, but updated against latest master changes and with all code factor issues in changed files resolved.

@stevenebutler stevenebutler mentioned this pull request Mar 15, 2023
Closed
@CarloToso CarloToso mentioned this pull request Mar 15, 2023
Closed
@adityapatwardhan
Copy link
Member

I think we should add tests for this behavior. I was thinking of executing Invoke-WebRequest and Invoke-RestMethod under the PowerShell API and call a Stop() on it. That should be equivalent to sending a cancellation.

Example of using the API:

$ps = [PowerShell]::Create()
$ps.AddScript("Invoke-WebRequest .......")
$ps.Stop()

and then use Wait-UntilTrue to verify that the pipeline was stopped in a resonable time.

adityapatwardhan
adityapatwardhan requested changes Mar 16, 2023
View reviewed changes
Copy link
Member
@adityapatwardhan adityapatwardhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add a test.

@ghost ghost added the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label Mar 16, 2023
@stevenebutler
Copy link
Contributor Author

I think we should add tests for this behavior. I was thinking of executing Invoke-WebRequest and Invoke-RestMethod under the PowerShell API and call a Stop() on it. That should be equivalent to sending a cancellation.

Example of using the API:

$ps = [PowerShell]::Create()
$ps.AddScript("Invoke-WebRequest .......")
$ps.Stop()

and then use Wait-UntilTrue to verify that the pipeline was stopped in a resonable time.

Ok, I'll look into it when I have some spare time. I assume there's a way to get the web server harness to delay writing, or should I modify it to permit that if it doesn't already?

@ghost ghost removed the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label Mar 16, 2023
@adityapatwardhan
Copy link
Member

There is a Delay in the WebListener -

PowerShell/test/tools/WebListener/Controllers/DelayController.cs

Lines 17 to 29 in 4767c8c

public class DelayController : Controller
{
public JsonResult Index(int seconds)
{
if (seconds > 0){
int milliseconds = seconds * 1000;
Thread.Sleep(milliseconds);
}
var getController = new GetController();
getController.ControllerContext = this.ControllerContext;
return getController.Index();
}

@stevenebutler
Copy link
Contributor Author

There is a Delay in the WebListener -

PowerShell/test/tools/WebListener/Controllers/DelayController.cs

Lines 17 to 29 in 4767c8c

public class DelayController : Controller
{
public JsonResult Index(int seconds)
{
if (seconds > 0){
int milliseconds = seconds * 1000;
Thread.Sleep(milliseconds);
}
var getController = new GetController();
getController.ControllerContext = this.ControllerContext;
return getController.Index();
}

The delay has to occur after headers are sent to test this change. That code looks like it may delay the response headers until after the delay. I can add something to delay after sending headers if that’s okay.

@adityapatwardhan
Copy link
Member

Sure, please add more functionality to WebListener if needed.

@iSazonov
Copy link
Collaborator

The PR does not solve the problem completely as I pointed in related issue.

8000
@stevenebutler
Copy link
Contributor Author

The PR does not solve the problem completely as I pointed in related issue.

I reviewed your comment in the related issue and I don't agree with your conclusion. You claim that cancellation tokens don't always work with the asynchronous API but the issue you link to does not support your argument. In that issue the core team dispute that cancellation tokens can't cancel the async APIs and provided some sample code that would solve the issue.

It looks like you want to use synchronous APIs with HttpClient and have it honour timeouts set on the HttpClient but were advised that the synchronous APIs are incomplete and won't do what you want. To resolve this issue we should be using the async APIs instead since they are fully cancellable (and if they're not then that's a legitimate bug the core team would fix if we can provide them with a reproducible test case).

If you have a specific test case that this PR doesn't solve for hanging on CTRL-C then it would be an oversight on my part (e.g. missing converting one of the synchronous reads to asynchronous with cancellation token), and I will look into it.

@stevenebutler
Copy link
Contributor Author

Sure, please add more functionality to WebListener if needed.

I have added a range of tests around the CTRL-C handling as suggested. I have also manually tried some of these cases using the /Stall endpoint in the WebListener and can confirm that once the download has started with existing PowerShell, the response cannot be cancelled if the stream has stalled.

With the changes, a stalled stream cancels immediately when CTRL-C is pressed.

I do have some concerns the tests may be a little timing sensitive since we are racing the CTRL-C with the response headers already having been received and there's no easy way to synchronise that. I've tried to keep the times really short so the tests don't take too long, but they could be increased if the tests are flaky in the CI system.

@iSazonov
Copy link
Collaborator

If you have a specific test case that this PR doesn't solve

You can see in related dotnet issue.

It looks like you want to use synchronous APIs with HttpClient

Web cmdlets are synchronous by design.
Asynchronous methods can be used for some reasons, but they do not make this code synchronous.
I already tried to do what you do and it is not a general solution and only complicates the code. All we need to do is stop the cmdlet and for this we don't need to implement CTS in the operations available to us because there are unavailable ones.

@stevenebutler
Copy link
Contributor Author

If you have a specific test case that this PR doesn't solve

You can see in related dotnet issue.

It looks like you want to use synchronous APIs with HttpClient

Web cmdlets are synchronous by design. Asynchronous methods can be used for some reasons, but they do not make this code synchronous. I already tried to do what you do and it is not a general solution and only complicates the code. All we need to do is stop the cmdlet and for this we don't need to implement CTS in the operations available to us because there are unavailable ones.

There do not appear to be any async versions of the methods that don't accept a CTS, and they were changed from synchronousOp(...) to asyncOp(..., cts).GetAwaiter.GetResult() in the patch because WebCmdlets are synchronous.

The PR includes tests that test the web cmdlets will cancel immediately while the connection is stalled for a variety of response types, including download to file, processing ATOM responses and processing JSON responses which definitely don't work with PS 7.3.3 and work with the PR.

The patch only makes an exception for using CTS when reading from memorystream, since they do not require IO so should terminate within a reasonable time.

If there's a specific case I've missed please describe it so the PR can be improved to address it (or rejected if it cannot be fixed).

@stevenebutler stevenebutler force-pushed the support-ctrl-c-in-invoke-webrequest-and-invoke-restmethod-when-connection-hangs branch 5 times, most recently from 5c48c2b to 0235558 Compare March 18, 2023 00:32
@stevenebutler stevenebutler removed the request for review from daxian-dbw March 18, 2023 00:32
@stevenebutler stevenebutler force-pushed the support-ctrl-c-in-invoke-webrequest-and-invoke-restmethod-when-connection-hangs branch from 1cc514c to ef95109 Compare March 21, 2023 23:59
@stevenebutler
Copy link
Contributor Author

@adityapatwardhan I'm not sure why it says I asked to remove the other review requests. I had requested you review the change again after I added unit tests for the feature as per your request. If those other people need to review as well, can you add them back to the PR reviewers?

@stevenebutler
Support CTRL-C when network disconnected
8e21076 
Uses async APIs with cancellation tokens to support cancellation throughout the download process.

Includes tests for stopping stalled downloads with different response types, HTTP vs HTTPS and compression streams.
@stevenebutler stevenebutler force-pushed the support-ctrl-c-in-invoke-webrequest-and-invoke-restmethod-when-connection-hangs branch from ef95109 to 8e21076 Compare March 26, 2023 02:06
@stevenebutler
Copy link
Contributor Author

Hi @adityapatwardhan - could you please provide some guidance on whether this PR is going to be reviewed again soon, or whether I should just close it because it is unsuitable?

@adityapatwardhan
Copy link
Member

@stevenebutler = Reviewing it today. sorry for the delay.

@ghost ghost added the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label Apr 3, 2023
@adityapatwardhan adityapatwardhan enabled auto-merge (squash) April 3, 2023 16:47
@adityapatwardhan adityapatwardhan added CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log and removed Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept labels Apr 3, 2023
@stevenebutler
Copy link
Contributor Author

Hi @adityapatwardhan , thank you for the review. I see the macos build has timed out and is blocking the merge. Is there something I should do to address this? It looks unrelated to my change.

Thanks for removing the additional tag.

@stevenebutler
Merge branch 'PowerShell:master' into support-ctrl-c-in-invoke-webreq…
9625fde 
…uest-and-invoke-restmethod-when-connection-hangs
@pull-request-quantifier-deprecated
Copy link

This PR has 285 quantified lines of changes. In general, a change size of upto 200 lines is ideal for the best PR experience!

Quantification details 

Label      : Large
Size       : +232 -53
Percentile : 68.5%

Total files changed: 10

Change summary by file extension:
.cs : +147 -53
.ps1 : +81 -0
.psm1 : +4 -0

Change counts above are quantified counts, based on the PullRequestQuantifier customizations.

Why proper sizing of changes matters

Optimal pull request sizes drive a better predictable PR flow as they strike a
balance between between PR complexity and PR review overhead. PRs within the
optimal size (typical small, or medium sized PRs) mean:

  • Fast and predictable releases to production:
    • Optimal size changes are more likely to be reviewed faster with fewer
      iterations.
    • Similarity in low PR complexity drives similar review times.
  • Review quality is likely higher as complexity is lower:
    • Bugs are more likely to be detected.
    • Code inconsistencies are more likely to be detected.
  • Knowledge sharing is improved within the participants:
    • Small portions can be assimilated better.
  • Better engineering practices are exercised:
    • Solving big problems by dividing them in well contained, smaller problems.
    • Exercising separation of concerns within the code changes.

What can I do to optimize my changes

  • Use the PullRequestQuantifier to quantify your PR accurately
    • Create a context profile for your repo using the context generator
    • Exclude files that are not necessary to be reviewed or do not increase the review complexity. Example: Autogenerated code, docs, project IDE setting files, binaries, etc. Check out the Excluded section from your prquantifier.yaml context profile.
    • Understand your typical change complexity, drive towards the desired complexity by adjusting the label mapping in your prquantifier.yaml context profile.
    • Only use the labels that matter to you, see context specification to customize your prquantifier.yaml context profile.
  • Change your engineering behaviors
    • For PRs that fall outside of the desired spectrum, review the details and check if:
      • Your PR could be split in smaller, self-contained PRs instead
      • Your PR only solves one particular issue. (For example, don't refactor and code new features in the same PR).

How to interpret the change counts in git diff output

  • One line was added: +1 -0
  • One line was deleted: +0 -1
  • One line was modified: +1 -1 (git diff doesn't know about modified, it will
    interpret that line like one addition plus one deletion)
  • Change percentiles: Change characteristics (addition, deletion, modification)
    of this PR in relation to all other PRs within the repository.

Was this comment helpful? 👍  :ok_hand:  :thumbsdown: (Email)
Customize PullRequestQuantifier for this repository.

Sorry, something went wrong.

@adityapatwardhan adityapatwardhan merged commit 17fb783 into PowerShell:master Apr 3, 2023
@adityapatwardhan
Copy link
Member

@stevenebutler Thank you for your contribution!

CarloToso pushed a commit to CarloToso/PowerShell that referenced this pull request Apr 8, 2023
@stevenebutler
Pr/19379 (#1)
1a6707d 
* Remove FormObject.cs and FormObjectCollection.cs (PowerShell#19383)

* Exclude redundant parameter aliases from completion results (PowerShell#19382)

* Revert "Remove FormObject.cs and FormObjectCollection.cs (PowerShell#19383)" (PowerShell#19387)

This reverts commit 190c99a.

* Add the parameter `-RelativeBasePath` to `Resolve-Path` (PowerShell#19358)

* Fix a crash in the type inference code (PowerShell#19400)

* Remove GetResponseObject (PowerShell#19380)

* Add `-Environment` parameter to `Start-Process` (PowerShell#19374)

* Add `-Environment` parameter to `Start-Process`

* address codefactor

* fix test for Windows

* handle case where value is $null to remove env var

* change variables to make it more clear what the test is doing

* Add PoolNames variable group to compliance pipeline (PowerShell#19408)

* Improve package management acceptance tests by not going to the gallery (PowerShell#19412)

* Skip VT100 tests on Windows Server 2012R2 as console does not support it (PowerShell#19413)

* Update the `ICommandPredictor` interface to reduce boilerplate code from predictor implementation (PowerShell#19414)

* Enable type conversion of `AutomationNull` to `$null` for assignment (PowerShell#19415)

* Remove code related to `#requires -pssnapin` (PowerShell#19320)

* Support CTRL-C when reading data and connection hangs for `Invoke-RestMethod` and `Invoke-WebRequest` (PowerShell#19330)

* Update to the latest NOTICES file (PowerShell#19332)

* Update the cgmanifest (PowerShell#19459)

* WIP: Harden default command test. (PowerShell#19416)
@daxian-dbw
Copy link
Member
daxian-dbw commented Apr 18, 2023 •
edited
Loading

@stevenebutler
Some of the Ctrl-C tests added in this PR are fragile and have been failing a lot in CIs, one example: https://github.com/PowerShell/PowerShell/pull/19529/checks?check_run_id=12840094279.

Invoke-WebRequest and Invoke-RestMethod support Cancellation through CTRL-C.Invoke-RestMethod: CTRL-C after stalled JSON download processes JSON response
Expected $true, but got $false.

Can you please take a look and make the tests more reliable? /cc @adityapatwardhan

@stevenebutler
Copy link
Contributor Author

@daxian-dbw The tests rely on timeouts occurring within about a second of when you'd normally expect them to occur which might be a bit risky if running on heavily subscribed hardware.

The only way I can think of to make them more reliable is to give a larger window for the passing of time vs the expectation that the cancellation will take affect. What is generally considered a reasonable test duration when testing for timeouts?

This was referenced Apr 18, 2023
Closed
Merged
@stevenebutler
Copy link
Contributor Author

Have created #19532 which increases timeouts but it is failing on macos due to an unrelated test right now.

@ghost
Copy link
ghost commented Apr 20, 2023

🎉v7.4.0-preview.3 has been released which incorporates this pull request.:tada:

Handy links:

@ghost ghost mentioned this pull request Apr 20, 2023
Closed
5 tasks
@iSazonov iSazonov mentioned this pull request Apr 26, 2023
Open
22 tasks
@TravisEz13 TravisEz13 added the CL-BreakingChange Indicates that a PR should be marked as a breaking change in the Change Log label May 22, 2023
@iSazonov iSazonov mentioned this pull request May 23, 2023
Merged
22 tasks
@stevenebutler stevenebutler deleted the support-ctrl-c-in-invoke-webrequest-and-invoke-restmethod-when-connection-hangs branch January 4, 2024 21:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adityapatwardhan adityapatwardhan adityapatwardhan approved these changes

Assignees

@adityapatwardhan adityapatwardhan

Labels
CL-BreakingChange Indicates that a PR should be marked as a breaking change in the Change Log CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log Large
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Invoke-WebRequest not responding to «Ctrc+c»
5 participants
@stevenebutler @adityapatwardhan @iSazonov @daxian-dbw @TravisEz13
0