Repository navigation
perf(dav): fetch federated address book cards with addressbook-multiget - #65942
SebastianKrupinski wants to merge 3 commits into
Conversation
|
/backport to stable35 |
|
/backport to stable34 |
|
/backport to stable33 |
428d8f9 to
d18ee25
Compare
0a4754c to
4524e68
Compare
DerDreschner
left a comment
There was a problem hiding this comment.
Four things I'd fix before merging (parse failures, PHP 8.2 backports, missing hrefs, batch size), plus two smaller ones, all inline.
Outside the diff: the commit has no Assisted-by trailer, although the AI box in the description is ticked.
Assisted by ClaudeCode:claude-opus-5-5
| assert(is_string($body)); | ||
|
|
||
| /** @var MultiStatus $multiStatus */ | ||
| $multiStatus = (new SabreXmlService())->expect('{DAV:}multistatus', $body); |
There was a problem hiding this comment.
A single resource the multiget can't carry now breaks the whole sync, on every run:
- A card with a control character VObject accepts, e.g. ESC (0x1B) in a display name. The remote writes
address-datainto the XML as is, the 207 body isn't well-formed, andexpect()throwsLibXMLException("PCDATA invalid Char value 27"). The per-card GET passed the raw bytes through. - A value larger than 10 MB:
SabreXmlServiceparses withoutLIBXML_PARSEHUGE, so an 11 MBaddress-data/calendar-datathrows "huge text node" (9 MB parses fine).
Either way the token isn't stored and the next run requests the same batch again. Falling back to per-resource requestGet() when a batch fails to parse would keep the sync moving.
There was a problem hiding this comment.
Since this only ever syncs another Nextcloud's system address book, and the cards are generated from user accounts.
-
10 MB values: account properties are capped at 2048 bytes in AccountManager, and the embedded avatar is a 512 px render. A single address-data can't get close to libxml's limit, so I don't think this can happen in practice.
-
Control characters: fair point, a display name with e.g. ESC (from LDAP, say) would produce a malformed multistatus. But the remote already sends that same broken XML to its own CardDAV clients syncing the system address book, so I think the right fix is sanitising account data on the remote, not a fallback in federation sync. I'd rather keep this PR focused. So I'd leave this as is for now.
| */ | ||
| abstract class ASyncService { | ||
| /** Maximum number of resources requested in one multiget REPORT */ | ||
| protected const int MULTIGET_BATCH_SIZE = 100; |
There was a problem hiding this comment.
Typed class constants need PHP 8.3. stable33 and stable34 still support PHP 8.2, so the requested backports would fail with a ParseError there. ASyncService is the parent of the CardDAV SyncService, so updateUser() would break as well. protected const MULTIGET_BATCH_SIZE = 100; works on all branches.
There was a problem hiding this comment.
This will be addressed in the backports, we should implement the newest syntax in main, and adapt it in a backport
| } | ||
|
|
||
| if ($changedResources !== []) { | ||
| foreach ($this->fetchCards($url, $absoluteUri, $userName, $sharedSecret, $changedResources) as $resource => $vCard) { |
There was a problem hiding this comment.
Cards the remote leaves out of the multiget response are skipped without a log line, and the token still advances, so they are never fetched again. Sabre's getMultipleNodes() simply omits children it can't load. The old GET threw on a 404, failed the run, and retried with the old token.
With --full this also deletes data: deletePendingCards() removes the local copy of a card that still exists on the remote. A remote with lagging read replicas is enough to trigger it (the sync report lists a new card, the multiget doesn't find it yet).
Fetching missing hrefs with requestGet() (and deleting the local card on a 404) would keep the old behaviour. At the least, they should be logged and the new token not returned.
There was a problem hiding this comment.
So I looked into this... but with the remote side in mind. Federated sync only ever talks to another Nextcloud, and only for the system address book, and that narrows this down quite a bit.
On the remote, SystemAddressbook::getChanges() already calls getChild() on every added or modified card, and reports any that can't be loaded as deleted. getMultipleChildren() applies the same filters. So a card that shows up as 200 in the sync report was readable when the report ran. It can only be missing from the multiget if something changes in the 1 sec between the report and pull.
I've added logging to track this failure.
| 'auth' => [$userName, $sharedSecret], | ||
| 'body' => $this->buildMultiGetRequestBody($hrefs, $reportName, $dataProperty), | ||
| 'headers' => ['Content-Type' => 'application/xml'], | ||
| 'timeout' => $this->config->getSystemValueInt( |
There was a problem hiding this comment.
A whole batch now has to arrive within one carddav_sync_request_timeout (30 s by default); before, each card had its own request. System address book cards embed the avatar as a base64 PHOTO, so 100 cards can be tens of MB, and 100 calendar objects with attachments (#65944) hundreds of MB, held in memory twice while parsing. On a slow link the same first batch times out on every run and the sync never moves forward. Limiting batches by size as well as count, or retrying with smaller batches on a timeout, would avoid that.
There was a problem hiding this comment.
I measured this on a test instance:
106 system address book cards averaging ~11 KB each (generated avatars), so a batch of 100 is about 1 MB. With uploaded photos, cards stay bounded by the 512 px avatar render. JPEG photos put a batch around 7–14 MB, and the extreme case of every user having a large PNG is maybe 40–70 MB. The first fits in 30 s on any reasonable server link. The second needs a very slow link and an unusual user base at the same time.
The calendar attachment case belongs to the calendar PR, not this one.
For slow links, carddav_sync_request_timeout already applies to the multiget, the same way it applies to the sync report, so admins can raise it without code changes. I'd rather not add size-based batching or retry logic for this edge case.
But I've added the carddav_sync_request_timeout setting to requestGet() also
| return false; | ||
| } | ||
|
|
||
| protected function requestGet( |
There was a problem hiding this comment.
Pre-existing: requestGet() doesn't pass carddav_sync_request_timeout, so it only gets the HTTP client's default of 30 s, unlike requestSyncReport() and requestMultiGet(). Raising the setting doesn't help the GET fallback.
There was a problem hiding this comment.
Out of scope but I added the setting to the requestGet().
| */ | ||
| private function fetchCards(string $url, string $addressBookUrl, string $userName, string $sharedSecret, array $hrefs): \Generator { | ||
| $reportName = '{' . Plugin::NS_CARDDAV . '}addressbook-multiget'; | ||
| if (!$this->requestReportSupport($addressBookUrl, $userName, $sharedSecret, $reportName)) { |
There was a problem hiding this comment.
Nit: this PROPFIND runs on every call with changes, so once per page of a truncated sync. Every Nextcloud remote loads the CardDAV plugin for addressbooks/ and supports addressbook-multiget, so the check could go (or be cached per run). Right now a failing probe also aborts the sync instead of falling back.
There was a problem hiding this comment.
The check is needed when a sync restarts. I would rater just keep it, it does very little to take it out or more it.
Signed-off-by: SebastianKrupinski <krupinskis05@gmail.com>
4524e68 to
5690b9a
Compare
…-multiget Signed-off-by: SebastianKrupinski <krupinskis05@gmail.com>
…-multiget Signed-off-by: SebastianKrupinski <krupinskis05@gmail.com>
SebastianKrupinski
left a comment
There was a problem hiding this comment.
Hey @DerDreschner
Thanks for the review.
I made two minor adjustments, the rest is not really relavent
| } | ||
|
|
||
| if ($changedResources !== []) { | ||
| foreach ($this->fetchCards($url, $absoluteUri, $userName, $sharedSecret, $changedResources) as $resource => $vCard) { |
There was a problem hiding this comment.
So I looked into this... but with the remote side in mind. Federated sync only ever talks to another Nextcloud, and only for the system address book, and that narrows this down quite a bit.
On the remote, SystemAddressbook::getChanges() already calls getChild() on every added or modified card, and reports any that can't be loaded as deleted. getMultipleChildren() applies the same filters. So a card that shows up as 200 in the sync report was readable when the report ran. It can only be missing from the multiget if something changes in the 1 sec between the report and pull.
I've added logging to track this failure.
| assert(is_string($body)); | ||
|
|
||
| /** @var MultiStatus $multiStatus */ | ||
| $multiStatus = (new SabreXmlService())->expect('{DAV:}multistatus', $body); |
There was a problem hiding this comment.
Since this only ever syncs another Nextcloud's system address book, and the cards are generated from user accounts.
-
10 MB values: account properties are capped at 2048 bytes in AccountManager, and the embedded avatar is a 512 px render. A single address-data can't get close to libxml's limit, so I don't think this can happen in practice.
-
Control characters: fair point, a display name with e.g. ESC (from LDAP, say) would produce a malformed multistatus. But the remote already sends that same broken XML to its own CardDAV clients syncing the system address book, so I think the right fix is sanitising account data on the remote, not a fallback in federation sync. I'd rather keep this PR focused. So I'd leave this as is for now.
| 'auth' => [$userName, $sharedSecret], | ||
| 'body' => $this->buildMultiGetRequestBody($hrefs, $reportName, $dataProperty), | ||
| 'headers' => ['Content-Type' => 'application/xml'], | ||
| 'timeout' => $this->config->getSystemValueInt( |
There was a problem hiding this comment.
I measured this on a test instance:
106 system address book cards averaging ~11 KB each (generated avatars), so a batch of 100 is about 1 MB. With uploaded photos, cards stay bounded by the 512 px avatar render. JPEG photos put a batch around 7–14 MB, and the extreme case of every user having a large PNG is maybe 40–70 MB. The first fits in 30 s on any reasonable server link. The second needs a very slow link and an unusual user base at the same time.
The calendar attachment case belongs to the calendar PR, not this one.
For slow links, carddav_sync_request_timeout already applies to the multiget, the same way it applies to the sync report, so admins can raise it without code changes. I'd rather not add size-based batching or retry logic for this edge case.
But I've added the carddav_sync_request_timeout setting to requestGet() also
| return false; | ||
| } | ||
|
|
||
| protected function requestGet( |
There was a problem hiding this comment.
Out of scope but I added the setting to the requestGet().
| */ | ||
| private function fetchCards(string $url, string $addressBookUrl, string $userName, string $sharedSecret, array $hrefs): \Generator { | ||
| $reportName = '{' . Plugin::NS_CARDDAV . '}addressbook-multiget'; | ||
| if (!$this->requestReportSupport($addressBookUrl, $userName, $sharedSecret, $reportName)) { |
There was a problem hiding this comment.
The check is needed when a sync restarts. I would rater just keep it, it does very little to take it out or more it.
Summary
Checklist
3. to review, feature component)stable32)AI (if applicable)