8000 WebClient needs to expose the timeout property (or improve Invoke-WebRequest implementation) when downloading large files · Issue #4679 · PowerShell/PowerShell · GitHub
[go: up one dir, main page]
Skip to content

WebClient needs to expose the timeout property (or improve Invoke-WebRequest implementation) when downloading large files #4679

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
swinster opened this issue Aug 26, 2017 · 10 comments
Closed

WebClient needs to expose the timeout property (or improve Invoke-WebRequest implementation) when downloading large files #4679

swinster opened this issue Aug 26, 2017 · 10 comments
Labels
Issue-Enhancement the issue is more of a feature request than a bug Resolution-Fixed The issue is fixed. WG-Cmdlets-Utility cmdlets in the Microsoft.PowerShell.Utility module
Milestone
Future

Comments

@swinster
Copy link
swinster commented Aug 26, 2017 •
edited
Loading

I have been having some issues interacting with REST full APIs, specifically with our product (Pexip) which is writing with Python and Django and uses Apache. Some of these issues are outlined in #2112 and #4274 and this involves a similar an linked problem. I also added a comment to https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-webrequest?view=powershell-5.1 that outlines this issue that I have repeated below:

Attempting to download a large file, GETting from a REST API.

The Basic Invoke-WebRequest

    Invoke-WebRequest -Method Get -Uri $resource -cred $cred -OutFile $Destination -TimeoutSec 1800

This would be the ideal. I can set timeout values and seemingly achieve what I want in one line. Unfortunately, this completely locked up the machine the script was running on. Basically, all the memory was consumed and after an hour or so, I had to end the script – of the 3.5 GB file, I think 200 MB was downloaded! What I needed is a way to stream the remote file to a local file without loading everything into memory.

Using WebClient

This seemed a lot better, however, the WebClient has no timeout property!!!

    $wc = New-Object System.Net.WebClient     
    $wc.Credentials = New-Object System.Net.NetworkCredential($User, $Pass)
    $wc.Headers.Add("Authorization", "Basic $encoded")
    $wc.DownloadFile($resource, $DestinationPath)

This is what I ended up using and in order to get the timeout to work, I used an extended class – luckily someone clever than I has achieved this for PowerShell using a C# class (https://koz.tv/setup-webclient-timeout-in-powershell/) although I assume the native classes in PS could also be used – but why?

I have also found that I must add the additional Authorisation Headers, otherwise the client will get challenged then end up download the file effectively twice. Using Fiddler and watching the Ethernet stats, you can see this happen. Indeed, this seemingly occurred for all methods

Using HTTP client.

Well I tried and failed dismally. However, I seem to create the Httpclient, it only ever seemed to stream the remote file into memory before dumping to a file. I know what I wanted to do, but do not have the wherewithal to do it.
@SteveL-MSFT SteveL-MSFT added this to the 6.1.0 milestone Aug 26, 2017
@SteveL-MSFT SteveL-MSFT added WG-Cmdlets-Utility cmdlets in the Microsoft.PowerShell.Utility module Issue-Enhancement the issue is more of a feature request than a bug labels Aug 26, 2017
@iSazonov
Copy link
Collaborator

@markekraus Is the Issue already fixed?

@markekraus
Copy link
Contributor

No. we added a resume feature but not a way to timeoul a download.

This is, however, something we could add. Since we do a looped read of the stream, we could insert logic in their to stop when the time out is reached.

@iSazonov
Copy link
Collaborator

Ah, I again confused these two PR.

@SteveL-MSFT SteveL-MSFT modified the milestones: 6.1.0-Consider, Future Jun 20, 2018
@CarloToso
Copy link
Contributor
CarloToso commented Mar 15, 2023 •
edited
Loading

@stevenebutler This issue can be fixed with the -ReceiveTimeout (the name is a placeholder) parameter discussed in #19315. I think you could fix this issue after we merge #19330

@iSazonov
Copy link
Collaborator

This issue touches several problems but the main request is "What I needed is a way to stream the remote file to a local file without loading everything into memory."

Does this problem still exist and in what specific scenarios?

@CarloToso
Copy link
Contributor
CarloToso commented Mar 16, 2023 •
edited
Loading

There are two problems here:

  • Download streaming. No longer an issue, Test: Invoke-WebRequest https://speed.hetzner.de/10GB.bin -OutFile 10gb.bin
    (it still writes to memory if we use -PassThru, but I think it's by design)
  • A timeout during the download. Can be addressed with a -ReceiveTimeout property

@iSazonov
Copy link
Collaborator
  • Download streaming. No longer an issue, Test: Invoke-WebRequest https://speed.hetzner.de/10GB.bin -OutFile 10gb.bin
    (it still writes to memory if we use -PassThru, but I think it's by design)

If so we should close the issue.

  • A timeout during the download. Can be addressed with a -ReceiveTimeout property

It is another request. I think it makes no sense. Otherwise we will have to add timeouts in all cmdlets like Copy-Item, Get-Content and so on. If you want to discuss this please open new issue.

@iSazonov iSazonov added the Resolution-Fixed The issue is fixed. label Mar 16, 2023
@ghost
Copy link
ghost commented Mar 17, 2023

This issue has been marked as fixed and has not had any activity for 1 day. It has been closed for housekeeping purposes.

@ghost ghost closed this as completed Mar 17, 2023
@andrei-epure-sonarsource
Copy link
andrei-epure-sonarsource commented Aug 17, 2023 •
edited
Loading

@CarloToso starting what version does the download streaming work by design, please?

Thanks!

@CarloToso
Copy link
Contributor

I tested it works correctly in 5.1, 7.2, 7.3, 7.4. I hypotesize it was fixed before 6.1, but I didn't check.

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Issue-Enhancement the issue is more of a feature request than a bug Resolution-Fixed The issue is fixed. WG-Cmdlets-Utility cmdlets in the Microsoft.PowerShell.Utility module
Projects
None yet
Milestone
Future
Development

No branches or pull requests
6 participants
@markekraus @swinster @SteveL-MSFT @iSazonov @andrei-epure-sonarsource @CarloToso
0