Repository navigation
fix(http): Stream responses with cURL when the stream option is set - #66759
DerDreschner wants to merge 2 commits into
Conversation
| public function extract(int $start, int $end): string { | ||
| fseek($this->source, $start); | ||
| return fread($this->source, $end - $start); | ||
| $data = stream_get_contents($this->source, $end - $start, $start); | ||
| if ($data === false) { | ||
| throw new Exception('Unable to read the source data'); | ||
| } | ||
| return $data; | ||
| } |
There was a problem hiding this comment.
This should not be changed, this has nothing to do with streaming
There was a problem hiding this comment.
It is related: the webcal refresh requests the feed with 'stream' => true (Connection.php:59) and passes the body straight to the importers. Until now IClient ignored that option and returned a fully downloaded stream. Now it's a real network stream, where fread() returns at most 8 KiB and feof() can come one read late. extract() would cut off larger events, and a truncated XML feed would go unnoticed.
Alternatively, we drop 'stream' => true in Connection, and the importers stay as they are.
There was a problem hiding this comment.
Then I made a mistake when updating the "webcal refresh". I am pretty sure making these changes will break this in the console.
Webcal refresh should download the contents to a temp file, then the importer uses fseek to read random parts from it
| $parsed = false; | ||
| do { | ||
| $chunk = fread($this->source, 4096); | ||
| // Finish at the first empty read, as network streams can report their end one read late | ||
| $final = $chunk === false || $chunk === ''; | ||
| // An empty source has no document to finish | ||
| if ($final && !$parsed) { | ||
| break; | ||
| } | ||
| $parsed = true; | ||
| if (!xml_parse($parser, $final ? '' : $chunk, $final)) { | ||
| throw new Exception( | ||
| xml_error_string(xml_get_error_code($parser)) | ||
| . ' At line: ' | ||
| . xml_get_current_line_number($parser) | ||
| ); | ||
| } | ||
| } | ||
| } while (!$final); |
There was a problem hiding this comment.
This should not be changed, this has nothing to do with streaming
There was a problem hiding this comment.
It is related: the webcal refresh requests the feed with 'stream' => true (Connection.php:59) and passes the body straight to the importers. Until now IClient ignored that option and returned a fully downloaded stream. Now it's a real network stream, where fread() returns at most 8 KiB and feof() can come one read late. extract() would cut off larger events, and a truncated XML feed would go unnoticed.
Alternatively, we drop 'stream' => true in Connection, and the importers stay as they are.
| public function extract(int $start, int $end): string { | ||
| fseek($this->source, $start); | ||
| return fread($this->source, $end - $start); | ||
| $data = stream_get_contents($this->source, $end - $start, $start); | ||
| if ($data === false) { | ||
| throw new Exception('Unable to read the source data'); | ||
| } | ||
| return $data; | ||
| } |
There was a problem hiding this comment.
This should not be changed, this has nothing to do with streaming
There was a problem hiding this comment.
It is related: the webcal refresh requests the feed with 'stream' => true (Connection.php:59) and passes the body straight to the importers. Until now IClient ignored that option and returned a fully downloaded stream. Now it's a real network stream, where fread() returns at most 8 KiB and feof() can come one read late. extract() would cut off larger events, and a truncated XML feed would go unnoticed.
Alternatively, we drop 'stream' => true in Connection, and the importers stay as they are.
Callers that request a response with "stream" => true read the body as if it were a local file. That only worked because IClient downloaded the whole body before returning. A network stream can return fewer bytes than requested, report its end one read late and throw transfer errors while it is read: - the calendar import reads each component with stream_get_contents() and finishes the XML document at the first empty read - the webcal refresh logs transfer errors that occur while importing - the DAV storage fills short reads with a RetryWrapper like other network storages - the link reference provider reads the page with stream_get_contents() - Imaginary downloads its preview into a sink, where transfer errors are handled Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: David Dreschner <david.dreschner@nextcloud.com>
IClient uses a plain CurlHandler, which ignores Guzzle's "stream" option: the whole body is downloaded into php://temp before get() returns. Requests with "stream" => true now go to a handler that runs the transfer on CurlMultiHandler and returns the response as soon as its body starts to arrive. The rest of the body is downloaded while it is read, closing it cancels the transfer, and it stays seekable because cURL writes into php://temp. Requests that also set a sink keep using the CurlHandler. The transfer keeps using cURL, so DNS pinning (CURLOPT_RESOLVE), HTTP/2 and all other cURL options still apply, unlike with Guzzle's StreamHandler. The timeout only counts the time spent waiting for the server, so readers that are slower than the timeout keep working. Like any network stream, a streamed body can return fewer bytes than requested and throws transfer errors while it is read, as IResponse::getBody() now documents. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: David Dreschner <david.dreschner@nextcloud.com>
8fb090b to
12b5f83
Compare
Summary
IClientignored'stream' => trueand always downloaded the whole body first. Streamed responses now return as soon as the body starts to arrive, and the rest downloads while it's read. They still use cURL, so DNS pinning and HTTP/2 keep working (unlike #52941).Like any network stream, a streamed body can return short reads and throw transfer errors while reading. The core callers that relied on the old behaviour are adapted.
TODO
IResponse::getBody()behaviour to app developersChecklist
3. to review, feature component)stable32)AI (if applicable)