Skip to content

fix(http): Stream responses with cURL when the stream option is set - #66759

Open
DerDreschner wants to merge 2 commits into
masterfrom
fix/make-stream-possible
Open

DerDreschner wants to merge 2 commits into
masterfrom
fix/make-stream-possible

Conversation

@DerDreschner

Copy link
Copy Markdown
Member

Summary

IClient ignored 'stream' => true and always downloaded the whole body first. Streamed responses now return as soon as the body starts to arrive, and the rest downloads while it's read. They still use cURL, so DNS pinning and HTTP/2 keep working (unlike #52941).

Like any network stream, a streamed body can return short reads and throw transfer errors while reading. The core callers that relied on the old behaviour are adapted.

TODO

  • Announce the changed IResponse::getBody() behaviour to app developers

Checklist

AI (if applicable)

  • The content of this PR was partly or fully generat

@DerDreschner
DerDreschner marked this pull request as ready for review November 18, 2026 11:45
@DerDreschner
DerDreschner requested review from Altahrim and icewind1991 and removed request for a team November 18, 2026 11:45
@DerDreschner DerDreschner self-assigned this Nov 18, 2026
@DerDreschner DerDreschner added 3. to review Waiting for reviews bug labels Nov 18, 2026
Comment on lines 153 to 159
public function extract(int $start, int $end): string {
fseek($this->source, $start);
return fread($this->source, $end - $start);
$data = stream_get_contents($this->source, $end - $start, $start);
if ($data === false) {
throw new Exception('Unable to read the source data');
}
return $data;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not be changed, this has nothing to do with streaming

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is related: the webcal refresh requests the feed with 'stream' => true (Connection.php:59) and passes the body straight to the importers. Until now IClient ignored that option and returned a fully downloaded stream. Now it's a real network stream, where fread() returns at most 8 KiB and feof() can come one read late. extract() would cut off larger events, and a truncated XML feed would go unnoticed.

Alternatively, we drop 'stream' => true in Connection, and the importers stay as they are.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then I made a mistake when updating the "webcal refresh". I am pretty sure making these changes will break this in the console.

Webcal refresh should download the contents to a temp file, then the importer uses fseek to read random parts from it

Comment on lines +62 to +79
$parsed = false;
do {
$chunk = fread($this->source, 4096);
// Finish at the first empty read, as network streams can report their end one read late
$final = $chunk === false || $chunk === '';
// An empty source has no document to finish
if ($final && !$parsed) {
break;
}
$parsed = true;
if (!xml_parse($parser, $final ? '' : $chunk, $final)) {
throw new Exception(
xml_error_string(xml_get_error_code($parser))
. ' At line: '
. xml_get_current_line_number($parser)
);
}
}
} while (!$final);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not be changed, this has nothing to do with streaming

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is related: the webcal refresh requests the feed with 'stream' => true (Connection.php:59) and passes the body straight to the importers. Until now IClient ignored that option and returned a fully downloaded stream. Now it's a real network stream, where fread() returns at most 8 KiB and feof() can come one read late. extract() would cut off larger events, and a truncated XML feed would go unnoticed.

Alternatively, we drop 'stream' => true in Connection, and the importers stay as they are.

Comment on lines 179 to 185
public function extract(int $start, int $end): string {
fseek($this->source, $start);
return fread($this->source, $end - $start);
$data = stream_get_contents($this->source, $end - $start, $start);
if ($data === false) {
throw new Exception('Unable to read the source data');
}
return $data;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not be changed, this has nothing to do with streaming

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is related: the webcal refresh requests the feed with 'stream' => true (Connection.php:59) and passes the body straight to the importers. Until now IClient ignored that option and returned a fully downloaded stream. Now it's a real network stream, where fread() returns at most 8 KiB and feof() can come one read late. extract() would cut off larger events, and a truncated XML feed would go unnoticed.

Alternatively, we drop 'stream' => true in Connection, and the importers stay as they are.

Callers that request a response with "stream" => true read the body as
if it were a local file. That only worked because IClient downloaded the
whole body before returning. A network stream can return fewer bytes
than requested, report its end one read late and throw transfer errors
while it is read:

- the calendar import reads each component with stream_get_contents()
  and finishes the XML document at the first empty read
- the webcal refresh logs transfer errors that occur while importing
- the DAV storage fills short reads with a RetryWrapper like other
  network storages
- the link reference provider reads the page with stream_get_contents()
- Imaginary downloads its preview into a sink, where transfer errors are
  handled

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: David Dreschner <david.dreschner@nextcloud.com>
IClient uses a plain CurlHandler, which ignores Guzzle's "stream"
option: the whole body is downloaded into php://temp before get()
returns. Requests with "stream" => true now go to a handler that runs
the transfer on CurlMultiHandler and returns the response as soon as
its body starts to arrive. The rest of the body is downloaded while it
is read, closing it cancels the transfer, and it stays seekable because
cURL writes into php://temp. Requests that also set a sink keep using
the CurlHandler.

The transfer keeps using cURL, so DNS pinning (CURLOPT_RESOLVE),
HTTP/2 and all other cURL options still apply, unlike with Guzzle's
StreamHandler. The timeout only counts the time spent waiting for the
server, so readers that are slower than the timeout keep working.

Like any network stream, a streamed body can return fewer bytes than
requested and throws transfer errors while it is read, as
IResponse::getBody() now documents.

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: David Dreschner <david.dreschner@nextcloud.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WebDAV external storage times out when browsing or syncing large files

3 participants