[HttpClient] Streaming request changes in HttplugClient result in curl error CURLE_SEND_FAIL_REWIND #59489

veewee · 2025-01-13T09:55:03Z

Symfony version(s) affected

7.2.2+

Description

Since this change in 7.2.2 that allows for streaming the HTTP request body with CURLOPT_READFUNCTION instead of CURLOPT_POSTFIELDS: bbdf0e0 (introduced by @nicolas-grekas)

We are receiving: CURLE_SEND_FAIL_REWIND (65)

When doing a send operation curl had to rewind the data to retransmit, but the rewinding operation failed.

  Unable to read stream contents: Necessary data rewind wasn't possible for "http://xxxx"

Exception trace:
  at /app/vendor/symfony/http-client/Response/CurlResponse.php:319
 Symfony\Component\HttpClient\Response\CurlResponse::perform() at /app/vendor/symfony/http-client/Response/TransportResponseTrait.php:167
 Symfony\Component\HttpClient\Response\CurlResponse::stream() at /app/vendor/symfony/http-client/Response/StreamWrapper.php:123
 Symfony\Component\HttpClient\Response\StreamWrapper->stream_read() at n/a:n/a
 stream_get_contents() at /app/vendor/nyholm/psr7/src/Stream.php:270
 Nyholm\Psr7\Stream->getContents() at /app/vendor/nyholm/psr7/src/StreamTrait.php:23
 Nyholm\Psr7\Stream->__toString() at /app/vendor/php-soap/psr18-transport/src/HttpBinding/Psr7Converter.php:41

This happens on the second request. The first request gets handled as expected.

The server is private and cannot be shared.
Some details:

It's a SOAP 1.1 server
It has NTLM authentication (meaning curl will perform additional NTLM authentication requests / responses internally)

This happens on:

PHP 8.3
curl version 8.11.0

Our best guess is that using CURLOPT_READFUNCTION in combination with NTLM is resulting in this specific error inside curl.
When we use this code-change on other endpoints, it works as expected.

We found following issues on the web which might or might not be related:

How to reproduce

As mentioned, the server is private and cannot be shared.

The configuration of the client is straight forward and looks like this:

use Symfony\Component\HttpClient\CurlHttpClient;
use Symfony\Component\HttpClient\HttplugClient;

$client = new HttplugClient(
    new CurlHttpClient([
        'auth_ntlm' => 'user:password',
    ])
)

ℹ️ The first request that is sent over the shared connection will succeed as usual. The second request will fail with the error Unable to read stream contents: Necessary data rewind wasn't possible

The request is a SOAP request that looks like this:

$headers = [
    0 => "Host: host:port"
    1 => "SOAPAction: "..."
    2 => "Content-Type: text/xml; charset="utf-8""
    3 => "Content-Length: 376"
    4 => "Accept: */*"
    5 => "User-Agent: Symfony HttpClient (Curl)"
    6 => "Accept-Encoding: gzip"
    7 => "expect:"
    8 => "Transfer-Encoding:"
];

$body = <<<EOXML
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
    <SOAP-ENV:Body xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
        <tns:AppInitialise xmlns:tns="uri:foo">
            <tns:useragent>foo</tns:useragent>
            <tns:languagecode>NLB</tns:languagecode>
        </tns:AppInitialise>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>
EOXML;

which results in following curl options:

^ array:31 [
  10002 => "http://..."
  121 => true
  181 => 3
  182 => 3
  52 => true
  68 => 20
  10031 => ""
  13 => 0
  10004 => null
  10177 => ""
  64 => true
  81 => 2
  10065 => null
  10097 => null
  10083 => null
  10025 => null
  10087 => null
  10026 => null
  172 => false
  32 => 6
  107 => 8
  84 => 2
  10005 => "user:password"
  91 => false
  229 => 1
  10036 => "POST"
  47 => true
  10023 => array:9 [
    0 => "Host: horst:port"
    1 => "SOAPAction: "action""
    2 => "Content-Type: text/xml; charset="utf-8""
    3 => "Content-Length: 376"
    4 => "Accept: */*"
    5 => "User-Agent: Symfony HttpClient (Curl)"
    6 => "Accept-Encoding: gzip"
    7 => "expect:"
    8 => "Transfer-Encoding:"
  ]
  20012 => Closure($ch, $fd, $length)^ {#39732
    class: "Symfony\Component\HttpClient\CurlHttpClient"
    use: {
      $body: Closure(int $size)^ {#39649 …}
      $eof: false
      $buffer: ""
    }
  }
  14 => 502
  10100 => CurlShareHandle {#39677}
]

Let me know if I can add anything to this issue report to make it easier for you to reproduce or figure out what is going wrong.

Possible Solution

We are not sure what is causing this specific error. It might be an issue in curl instead of symfony/http-client.

Following configuration got us around the actual issue, but seems like a bad idea performance-wise.

CURLOPT_FORBID_REUSE - make connection get closed at once after use

$client = new HttplugClient(
    new CurlHttpClient([
        'auth_ntlm' => 'user:password',
        'extra' => [
            'curl' => [
                \CURLOPT_FORBID_REUSE => true,
            ]
        ]
    ])
);

Another option around the problem would be to introduce a new CurlHttpClient option.
Something like $options['stream_request'] which has a default of true to use the CURLOPT_READFUNCTION implementation, but can be set to false for using the CURLOPT_POSTFIELDS logic.

Additional Context

No response

The text was updated successfully, but these errors were encountered:

nicolas-grekas · 2025-01-13T10:30:20Z

Linked commit comes from #58856

This exception reminds me #58901
Can you please try if this patch fixes the issue?

veewee · 2025-01-13T10:49:20Z

Thanks for your swift response!

That change does not seem to fix the error though:

Rewinding the request body works as expected. The error mentioned in this issue is happening at curl level:

Request1 -> curl -> Response1
Request -> curl (throws CURLE_SEND_FAIL_REWIND)

It seems to be the internal curl stream that cannot be rewind properly.

I've also found this debug information inside CurlResponse

          error: "Necessary data rewind wasn't possible for "http://xxxx?wsdl"."
          debug: """
            * Found bundle for host: 0xaaaaaaaaaaaaaa [serially]\n
            * Re-using existing connection #0 with host host\n
            * Server auth using NTLM with user 'user'\n
            > POST /xxx?wsdl HTTP/1.1\r\n
            Host: host:port\r\n
            SOAPAction: "foo"\r\n
            Content-Type: text/xml; charset="utf-8"\r\n
            Content-Length: 500\r\n
            Accept: */*\r\n
            User-Agent: Symfony HttpClient (Curl)\r\n
            Accept-Encoding: gzip\r\n
            \r\n
            < HTTP/1.1 401 Unauthorized\r\n
            < Content-Length: 0\r\n
            < Server: Microsoft-HTTPAPI/2.0\r\n
            * NTLM auth restarted\n
            < WWW-Authenticate: NTLM\r\n
            < Date: Mon, 13 Jan 2025 10:41:35 GMT\r\n
            * Rewind stream before next send\n
            * Keep sending data to get tossed away\n
            < \r\n
            * Connection #0 to host host left intact\n
            * Issue another request to this URL: 'http://host/x?wsdl'\n
            * Found bundle for host: 0xaaaaaaaaa [serially]\n
            * Re-using existing connection #0 with host host\n
            * necessary data rewind wasn't possible\n
            * Closing connection 0\n
            """

(Note : the 401 is ntlm asking to reauthenticate which is handled by curl, the credentials are valid)

nicolas-grekas · 2025-01-13T10:56:29Z

OK, so basically we'd need to implement https://curl.se/libcurl/c/CURLOPT_SEEKFUNCTION.html but this isn't possible in PHP...

8000

I'm wondering if this would help?

--- a/src/Symfony/Component/HttpClient/Psr18Client.php
+++ b/src/Symfony/Component/HttpClient/Psr18Client.php
@@ -100,7 +100,7 @@ final class Psr18Client implements ClientInterface, RequestFactoryInterface, Str
 
             $options = [
                 'headers' => $headers,
-                'body' => static fn (int $size) => $body->read($size),
+                'body' => $body->detach(),
             ];

veewee · 2025-01-13T11:19:34Z

This change doesn't work either:

            * Found bundle for host: 0xaaaaeaaaa [serially]\n
            * Re-using existing connection #0 with host host\n
            * Server auth using NTLM with user 'userr'\n
            > POST foo?wsdl HTTP/1.1\r\n
            Host: host:port\r\n
            Transfer-Encoding: chunked\r\n
            SOAPAction: "action"\r\n
            Content-Type: text/xml; charset="utf-8"\r\n
            Content-Length: 504\r\n
            Accept: */*\r\n
            User-Agent: Symfony HttpClient (Curl)\r\n
            Accept-Encoding: gzip\r\n
            \r\n
            * Signaling end of chunked upload via terminating chunk.\n
            < HTTP/1.1 401 Unauthorized\r\n
            < Content-Length: 0\r\n
            < Server: Microsoft-HTTPAPI/2.0\r\n
            * NTLM auth restarted\n
            < WWW-Authenticate: NTLM\r\n
            < Date: Mon, 13 Jan 2025 11:13:45 GMT\r\n
            * Rewind stream before next send\n
            * Keep sending data to get tossed away\n
            < \r\n
            * Connection #0 to host host left intact\n
            * Issue another request to this URL: 'http://service?wsdl'\n
            * Found bundle for host: 0xaaaaeaaaaaa [serially]\n
            * Re-using existing connection #0 with host host\n
            * necessary data rewind wasn't possible\n
            * Closing connection 0\n

This dump info contains the previous request basically. I think you're right: when using either CURLOPT_INFILE or CURLOPT_READFUNCTION the curl stream needs to get seeked first.
I don't really understand how all of this works, but it's strange that it does work when using CURLOPT_POSTFIELDS directly...

nicolas-grekas · 2025-01-22T15:33:53Z

Before the #58856, the content was give as a string to curl. A string it easy to rewind so curl just did it.

Not that we give it a read function, it cannot rewind anymore.

$body->detach() should give curl a stream, and those are rewindable. You might want to dump($body->detach()) to see what kind of stream we're passing.

It's strange to me that both Transfer-Encoding and Content-Length are sent in your last example. Dunno if that's related.

Looks like NTLM is a mess. :D

Following configuration got us around the actual issue, but seems like a bad idea performance-wise.
CURLOPT_FORBID_REUSE - make connection get closed at once after use

Actually, looking at the behavior we have here, I'd say this might be more performant than figuring out how we could rewind the stream, since apparently reusing connections requires sending a request, having it fail and have curl retry.

nicolas-grekas · 2025-01-22T15:34:49Z

If you run some bechmarks maybe and help us decide if we should make CURLOPT_FORBID_REUSE the default when auth_ntlm is used?

veewee · 2025-01-23T10:26:21Z

@nicolas-grekas

$body->detach() should give curl a stream, and those are rewindable. You might want to dump($body->detach()) to see what kind of stream we're passing.

It's a regular Nyholm\Psr7\Stream which is rewindable. This request body stream is in any case not what is causing the issue. (You'll see later in my benchmark that I am reusing the request)

It's strange to me that both Transfer-Encoding and Content-Length are sent in your last example. Dunno if that's related.

I was playing with those things cause that was my initial hunch as well. The pasted example is not correct. It actually uses Content-Length with empty Transfer-Encoding.

Looks like NTLM is a mess.

Don't agree there, but it's also not the worst thing I've seen in Microsoft ... :D

Actually, looking at the behavior we have here, I'd say this might be more performant than figuring out how we could rewind the stream, since apparently reusing connections requires sending a request, having it fail and have curl retry.
If you run some bechmarks maybe and help us decide if we should make CURLOPT_FORBID_REUSE the default when auth_ntlm is used?

Wow ... This is where things become weird:

So my benchmark is using http-client 7.2.2.
I run it 2 times:

first time with the diff excluded, meaning it uses a raw string throuhgCURLOPT_POSTFIELDS
second time with the change that uses CURLOPT_READFUNCTION, with an additional option of CURLOPT_FORBID_REUSE

The benchmark looks like this:

use Symfony\Component\HttpClient\CurlHttpClient;
use Symfony\Component\HttpClient\HttplugClient;
use Symfony\Component\Stopwatch\Stopwatch;

$_ENV['SERVICE_ATTEMPTS'] = 20;


$client = $client = new HttplugClient(new CurlHttpClient([
    'auth_ntlm' => [$_ENV['SERVICE_USER'], $_ENV['SERVICE_PASS']],
    'extra' => [
        'curl' => [
            \CURLOPT_FORBID_REUSE => true,
        ]
    ]
]));

$request = $client->createRequest('POST', 'foo')
    ->withHeader('SOAPAction', '"doesnotexist"')
    ->withHeader('Content-Type', 'text/xml; charset="utf-8"')
    ->withBody($client->createStream(<<<EOXML
    <xxx />
    EOXML
));

dump('Sending '.$_ENV['SERVICE_ATTEMPTS'].' request...');

$stopWatch = new Stopwatch(true);
$stopWatch->start('request');

try {
    for ($i=0; $i<$_ENV['SERVICE_ATTEMPTS']; $i++) {
        dump('Attempt '.$i);

        $response = $client->sendRequest($request);
        $stopWatch->lap('request');

        echo $response->getBody() . PHP_EOL;
    }
} catch (\Exception $e) {
    dump($e->getMessage());
    dd($e->getTraceAsString());
}


$event = $stopWatch->stop('request');

dump('total duration (ms): '.$event->getDuration());
dump('AVG duration (ms): '.$event->getDuration()/$_ENV['SERVICE_ATTEMPTS']);

The results are strange to me:

CURLOPT_POSTFIELDS (without CURLOPT_FORBID_REUSE) has "AVG duration (ms): 208.9"
CURLOPT_POSTFIELDS (with CURLOPT_FORBID_REUSE) has "AVG duration (ms): 190"
CURLOPT_READFUNCTION with CURLOPT_FORBID_REUSE has an "AVG duration (ms): 531.725"
CURLOPT_READFUNCTION without CURLOPT_FORBID_REUSE breaks after request 1, but also seems to have around 5xx ms.

I've tried the same flow a couple of times on different period to make sure the results are representative and not too affected by the server response time. I also choose to send a non-existing soap method to get an instant soap error back, to make sure the server logic hasn't too much of an impact either.

So I can only conclude that:

the FORBID_REUSE doesn't have a big impact on performance (meaning it could be set by-default when NTLM is active)
BUT somehow streaming the body seems more than double slower than sending the same payload as a string.

If streaming the body is soo much slower for the same request, wouldn't it make more sense to:

look into why streaming is considerably slower.
- If it can be fixed : nice!
- if not it should be something to manually opt-in to or automatically opt-in if the stream size is bigger than the memory available?

I've debugged the buffer size that is provided to the function and it is 65536 bytes. That is far beyond the size of my request payload.

nicolas-grekas · 2025-01-23T14:07:44Z

It's a regular Nyholm\Psr7\Stream which is rewindable

That's not possible : detach() returns a resource, not a PSR7 object.

Thanks for the benchmarks! So let's set CURLOPT_FORBID_REUSE when auth_ntlm is in use. Up for a PR doing so?

This makes me think: can you also benchmark with using $body->detach() instead of the current close?

veewee · 2025-01-23T14:35:48Z

That's not possible : detach() returns a resource, not a PSR7 object.

It's a seekable memory stream internally.

This makes me think: can you also benchmark with using $body->detach() instead of the current close?

It's similar to CURLOPT_READFUNCTION.

Thanks for the benchmarks! So let's set CURLOPT_FORBID_REUSE when auth_ntlm is in use. Up for a PR doing so?

That is possible and would fix the error indeed. The performance overhead is something that is keeping me reluctant to choosing that solution though. It's consitently double as slow on this service ... What's your thought on that topic?

nicolas-grekas · 2025-01-23T15:17:55Z

If CURLOPT_FORBID_REUSE fixes the issue, I'd still do it, regardless of the perf issue.
About the perf issue, some profiling would be needed do understand what's going on...

nicolas-grekas · 2025-01-29T15:08:50Z

I submitted the fix in #59648 by:

using a string body for bodies < 2MB
forbidding connection reuse for non-string bodies

veewee · 2025-01-29T15:31:51Z

Thanks. I've applied the patch on HttplugClient as well and it seems to work perfectly without any stream issues nor loss in performance. 👍

…NTLM connections (nicolas-grekas, ajgarlag) This PR was merged into the 7.2 branch. Discussion ---------- [HttpClient] Fix retrying requests with `Psr18Client` and NTLM connections | Q | A | ------------- | --- | Branch? | 7.2 | Bug fix? | yes | New feature? | no | Deprecations? | no | Issues | Fix #59489, Fix #59629 | License | MIT Commits ------- b647cdf Prevent empty request body stream in HttplugClient and Psr18Client f0239d8 [HttpClient] Fix retrying requests with Psr18Client and NTLM connections

veewee added the Bug label Jan 13, 2025

carsonbot added Status: Needs Review HttpClient labels Jan 13, 2025

veewee changed the title ~~[HttpClient]~~ [HttpClient] Streaming request changes in HttplugClient result in curl error CURLE_SEND_FAIL_REWIND Jan 13, 2025

nicolas-grekas mentioned this issue Jan 29, 2025

[HttpClient] Fix retrying requests with Psr18Client and NTLM connections #59648

Merged

nicolas-grekas closed this as completed Feb 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[HttpClient] Streaming request changes in HttplugClient result in curl error CURLE_SEND_FAIL_REWIND #59489

[HttpClient] Streaming request changes in HttplugClient result in curl error CURLE_SEND_FAIL_REWIND #59489

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[HttpClient] Streaming request changes in HttplugClient result in curl error CURLE_SEND_FAIL_REWIND #59489

[HttpClient] Streaming request changes in HttplugClient result in curl error CURLE_SEND_FAIL_REWIND #59489

Comments

Uh oh!

Symfony version(s) affected

Description

How to reproduce

Possible Solution

Additional Context

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!