[feature](file cache) Support peer cache reads on the async cache write path by deardeng · Pull Request #69071 · apache/doris

deardeng · 2026-10-14T11:27:31Z

What problem does this PR solve?

Issue Number: None

Related PR: #65658

Problem Summary:

The async file cache write path (#65658) never read from peers: with peer read enabled, _resolve_cache_write_mode fell back to SYNC_WRITE, and the async path forced bypass_peer_read. So a cluster could use async write (for slow cache disks) or peer cache, but not both.

This PR lets the async path read its misses from peers. Peer data stays zero-copy: the RPC attachment IOBuf goes into the inflight index and is persisted with FileBlock::append_iobuf; the only copy is into the user buffer.

Peer chain takes ranges. PeerFileCacheReader::fetch_blocks and the peer dispatch helpers take std::vector<FileBlock::Range> instead of FileBlockSPtr, because the async path has no FileBlock for its misses.
IOBuf-backed async write buffers. AsyncCacheWriteManager::wrap_iobuf() wraps one peer block without copying and charges it to the async-write MemTracker.
Async path reads from peers, same span rules as the sync path:
- a span of pure misses is sent to peer as is;
- cached / inflight blocks between misses are read locally, and peer is asked only for the misses;
- a span with a DOWNLOADING block, or whose tablet has no usable peer candidate, reads from storage as before.
Race S3 pool size. PeerRaceS3ThreadPool runs the S3 leg of every peer/S3 race (sync and async). It had 32 threads for 64 race slots and an unbounded queue; it is now sized max(max_peer_race_s3_thread_num, max_concurrent_peer_races), 64 by default.
Fixes along the way.
- BytesReadFrom{Cross,Same}CGPeer counted the whole span instead of the bytes peer returned, which also overcounted on the sync path.
- The race lambda copied its arguments twice; now it copies once.
Observability.
- Profile: AsyncCopyToUserTimer, AsyncSubmitWriteTimer.
- Bvars: cached_remote_file_reader_async_peer_{eligible,no_candidate,bypass}_span_count.

Performance

Cold full scan of SSB SF1000 lineorder (6.0 B rows, 17 columns, 135.8 GB). One source BE with a warm cache and one reader BE, in different compute groups. Default race config. Medians of 3 runs.

A: sync write + peer
B: async write + S3 (async write before this PR)
C: async write + peer (this PR)

Source → reader cache disk	Scanners	A	B	C	C vs B	C vs A	S3 GETs, C vs B
NVMe → PL1	64	216.7 s	100.7 s	43.8 s	−57%	−80%	−91%
NVMe → PL1	128	217.2 s	55.2 s	44.2 s	−20%	−80%	−90%
NVMe → NVMe ¹	64	42.6 s	104.8 s	44.9 s	−57%	+6%	−99.9%
NVMe → NVMe ¹	128	43.1 s	60.6 s	45.6 s	−25%	+6%	−99.9%
PL1 → PL1	64	222.8 s	100.7 s	127.3 s	+27%	−43%	−28%
PL1 → PL1	128	–	56.1 s	111.9 s	+99%	–	−32%

¹ Tuned race config (256 slots / 256 S3 threads), not rerun at the default.

C beats B when the peer source is faster than S3. With a PL1 source (~0.5 GB/s) peer is slower than S3. Most of that loss comes from the race config: with a larger S3 pool and no hedge delay, C matches B.
C beats A when the reader's cache disk is slow. On an NVMe reader, A pays almost nothing for the cache write, so async write gives no gain there.
Cost: less cache retained. On a PL1 reader the async path drops pending writes it can't persist in time. After one scan C kept 54 GiB, B kept 80 GiB, A kept 136 GiB. Running the same query again was still faster with C (32.9 s vs B 52.7 s), because C refilled its misses from peer.
Limits:
- These runs used the old default pool of 32 threads.
- C relied on the reader's network burst bandwidth.
- Throughput plateaus around 3 GB/s, probably reader CPU; the cause is not isolated.
- Not tested: a no-regression run against master with peer off, sparse-read performance, small reads, long-running load, more than one source BE.

Method and race-config evidence

Machines. ecs.i5.8xlarge with local NVMe; ecs.g9i.8xlarge with 2 × 300 GiB ESSD PL1. Same zone.
Before each cold run:
- clear the reader's file cache;
- drop_caches on the source (not done in the PL1 → PL1 round);
- storage page cache off on both BEs.
Settings. Pending write budget 4 / 12 GiB at 64 / 128 scanners. Binary at the commit before the pool change.
Race S3 pool (PL1 → PL1, async write + peer, 20 ms hedge):

Scanners	Slots / S3 threads	Time	S3 pool queue peak	Wait per S3 task
64	64 / 32	169.2 s	16	14.9 ms
64	256 / 256	144.3 s (−15%)	0	0.009 ms
128	64 / 32	124.2 s	19	16.9 ms
128	256 / 32	172.5 s (+39%)	103–264	~110 ms
128	256 / 256	77.4 s (−55% vs above)	0–1	0.017 ms

64 scanners: the slots were not the limit (peak 63 races, 0 serial peer reads), so the gain comes from the pool alone.
128 scanners: raising only the slots made it slower. Raising the pool as well fixed it.
Hedge delay (same setup, 256 / 256): 20 ms → 0 ms took C from 144.3 s to 99.2 s at 64 scanners and from 77.4 s to 57.2 s at 128, which is on par with B. Hedge 0 is not proposed here, because with a fast peer it doubles reader traffic.
Follow-up (not in this PR): after the peer wins, an S3 leg still sends its GET once it leaves the queue.

Release note

The async file cache write path (enable_async_file_cache_write) now reads missing blocks from peer BEs when enable_cache_read_from_peer is enabled. PeerRaceS3ThreadPool now uses at least max_concurrent_peer_races threads (64 by default, was 32). New profile counters: AsyncCopyToUserTimer, AsyncSubmitWriteTimer. New bvars: cached_remote_file_reader_async_peer_eligible_span_count, cached_remote_file_reader_async_peer_no_candidate_span_count, cached_remote_file_reader_async_peer_bypass_span_count.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
  - Cluster benchmark, see Performance.
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
  - With both async write and peer read enabled, cache misses now use the async path and read from peers, instead of falling back to sync write.
  - The race S3 pool defaults to 64 threads instead of 32.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-10-14T11:27:37Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

deardeng · 2026-10-14T11:28:11Z

/review

github-actions

Reviewed head 5898e994c44a132719bd9751b9c3c00d306f251f across all 16 changed files. No verified new findings and no existing inline blockers. The static review converged in 1 round: all three assigned reviewers returned NO_NEW_VALUABLE_FINDINGS, and all initial risk items received independently checked dispositions.

Critical checkpoint conclusions:

Goal and evidence: the async path now reads peer misses, returns the validated peer bytes, and queues shared IOBuf payloads for persistence. Added unit tests cover this behavior; they were inspected, not executed here.
Scope and reuse: the changes remain focused on async peer support and its profiles/tests. Shared remote dispatch and buffer copy/persistence interfaces are reused.
Concurrency and locks: query readers, peer bthreads, S3 pool tasks and write workers were traced. Existing race, cache, queue and index locks protect shared state; disk IO and eviction finalizers stay outside the queue lock. No new lock inversion found.
Lifecycle and memory: request ranges/context captures and reader ownership survive late race completion. Immutable IOBuf wrappers remain owned by tasks/inflight readers; rejection, eviction, stale epochs and shutdown preserve cleanup. Payload charges balance at wrapper destruction; they are logical bytes rather than exact backing-slab/RSS measurements. No new SIOF or ownership cycle found.
Configuration: no config added; existing mode/peer settings are read per call. Warmup/dryrun precedence and mixed-source peer bypass remain explicit.
Compatibility: internal callers use the new range interface consistently. No disk-format or new libc/dependency change identified; protobuf responses remain supported for older peers. Final release ELF compatibility was not audited.
Thrift/protobuf field IDs: not applicable; no definitions changed.
Parallel paths: sync/async, sequential/race, peer/storage, attachment/protobuf, cache/inflight/downloading, prefetch/warmup and remote-only paths were checked.
Special conditions and correctness: exact-coverage parsing, original block indexes, sorted regrouping, user-range clipping and EOF write_size support the new assertions. The mixed-hit and whole-range fallback conditions explain their concrete scenarios. No new data-corruption route found.
Tests: positive and negative cases cover peer success/reorder, miss/error/incomplete fallback, mixed-hit bypass, both race outcomes, multi-segment buffers, short EOF persistence, eviction tracker release and profile aggregation.
Test results: no regression goldens changed. No builds or unit/regression tests ran; the author's ASAN results were not treated as executed evidence from this review.
Observability: both new timers propagate through read statistics, merge/delta reporting, scanner detection, profile counters and the FileCache total. Existing source counters and worker error logs remain available.
Transactions, EditLog and failover: no changes to transaction visibility, partition versions, MoW/delete bitmaps or FE metadata persistence.
Cache writes and crash behavior: original miss bits, valid EOF sizes, downloader ownership, epoch fences and finalize-before-publication are preserved. Persistence failure remains opportunistic and does not invalidate successful query bytes.
FE-BE variables: not applicable; none added.
Performance and BE hygiene: peer tasks share IOBuf payloads and reuse append_iobuf without flattening; task creation adds no per-block payload copy after peer parsing, and user bytes are copied into the result buffer. The read-only bash build-support/check-build-hygiene.sh passed all checks. No measured performance claim made.
Other concerns and user focus: no additional user focus was supplied. The initial coverage/index, buffer ownership/accounting and late-race lifecycle concerns were closed with code evidence; no issue was withheld or duplicated.

Validation limits: no runtime sanitizer, OOM, exact RSS or load/scheduling measurements were performed. Dedicated async-peer inflight/EOF integration, active-read policy changes, shutdown/invalidation under load and memory-storage retention do not have newly executed proof here. The peer-win tests use an existing timing heuristic, so deterministic scheduling is not claimed. No unresolved PR-specific static candidate or omitted changed-file scope remains.

Review converged after 1 round(s).

…hen misses surround local blocks ### What problem does this PR solve? Issue Number: N/A Related PR: apache#69071 Problem Summary: A peer fails the whole RPC when it misses any requested block. So on the async cache write read path, a remote span with cached or inflight blocks between its misses (miss-hit-miss) bypassed peer and was reread entirely from remote storage, while the sync path requests only its empty blocks from peer and reads the blocks in between locally. Align the async path with the sync path. When peer read applies and the non-miss blocks inside the span are all CACHE or INFLIGHT: - Those blocks are read locally first with `_materialize_async_block` (CACHE blocks are touched in the LRU). Their bytes count as local only when all of them succeed. - Peer receives only the miss ranges. On a peer win the misses are copied into the result and their IOBufs are placed at the misses' span positions, so `_submit_async_write_tasks` is unchanged. - A storage fallback (no candidate, peer failure, or S3 winning the race) still reads the whole span, from which only the misses are copied. The span keeps the existing whole-span storage read in three cases: it contains a DOWNLOADING block (the async path never waits on another downloader), it is the whole-range fallback after a side race (marked on the plan), or a local read inside the span fails; a NOT_FOUND there still triggers self-heal. When peer read does not apply, the path is unchanged. Two bvars report how often peer is skipped because of the span shape: - `cached_remote_file_reader_async_peer_eligible_span_count`: async-path remote spans that may use peer read (peer read enabled and not bypassed by the caller). - `cached_remote_file_reader_async_peer_bypass_span_count`: eligible spans that read remote storage only because of one of the three shape cases above. Spans that try peer but end on remote storage (no candidate, peer miss or failure, S3 winning the race) are not counted. ### Release note None ### Check List (For Author) - Test: Unit Test - New CachedRemoteFileReaderAsyncPeerTest cases: miss-cached-miss with peer winning (peer gets two ranges, middle read locally, two writes submitted, second read local); the same span with peer failing (storage supplies only the misses, no double counting); an inflight middle block; a NOT_FOUND middle block rereads the span and self-heals; a downloading middle block and the whole-range fallback bypass peer. - New CrossCGAsyncWinnerRaceTest cases: sparse peer wins the race; an eligible span without candidates copies only the misses from storage. - Split and reordered peer responses: the mock peer service returns every requested block as uneven pieces, interleaved and out of order, for a sparse span and for an all-miss span, in attachment and protobuf-bytes modes. Each case checks the result bytes, the peer/local/remote byte counts, and the cache blocks read back directly after the async writes. With the chunk sort in take_peer_span removed for a local experiment (not committed), all four fail only on the cache read-back ("LIJK", "GHEF", "KLIJ" instead of the peer bytes) while the result bytes stay correct. - Before the change the new sparse cases fail (no peer RPC, whole span from storage, no self-heal for the middle block) and the counter cases fail to link. - CachedRemoteFileReaderPeerTest.*, CachedRemoteFileReaderAsyncPeerTest.*, CrossCGWinnerRaceTest.*, CrossCGAsyncWinnerRaceTest.*, AsyncCachedRemoteFileReaderTest.*, BlockFileCacheTest.*, AsyncCacheWriteManagerTest.*, InflightWriteBufferIndexTest.*, FileCacheProfileReporterTest.*: 276 tests, 273 pass and 3 pre-existing DISABLE_ tests are skipped (ASAN). The 20 async peer and race cases pass 10 repeated runs. - Unit tests were run locally with GLIBC_COMPATIBILITY=OFF; there is no performance data yet. - Behavior changed: Yes. With async cache write and peer read enabled, a span whose misses surround cached or inflight blocks reads those blocks locally and requests only the misses from peer. - Does this need documentation: No

…hen misses surround local blocks ### What problem does this PR solve? Issue Number: N/A Related PR: apache#69071 Problem Summary: A peer fails the whole RPC when it misses any requested block. So on the async cache write read path, a remote span with cached or inflight blocks between its misses (miss-hit-miss) bypassed peer and was reread entirely from remote storage, while the sync path requests only its empty blocks from peer and reads the blocks in between locally. Align the async path with the sync path. When peer read applies and the non-miss blocks inside the span are all CACHE or INFLIGHT: - Those blocks are read locally first with `_materialize_async_block` (CACHE blocks are touched in the LRU). Their bytes count as local only when all of them succeed. - Peer receives only the miss ranges. On a peer win the misses are copied into the result and their IOBufs are placed at the misses' span positions, so `_submit_async_write_tasks` is unchanged. - A storage fallback (no candidate, peer failure, or S3 winning the race) still reads the whole span, from which only the misses are copied. The span keeps the existing whole-span storage read in three cases: it contains a DOWNLOADING block (the async path never waits on another downloader), it is the whole-range fallback after a side race (marked on the plan), or a local read inside the span fails; a NOT_FOUND there still triggers self-heal. When peer read does not apply, the path is unchanged. Two bvars report how often peer is skipped because of the span shape: - `cached_remote_file_reader_async_peer_eligible_span_count`: async-path remote spans that may use peer read (peer read enabled and not bypassed by the caller). - `cached_remote_file_reader_async_peer_bypass_span_count`: eligible spans that read remote storage only because of one of the three shape cases above. Spans that try peer but end on remote storage (no candidate, peer miss or failure, S3 winning the race) are not counted. Peer byte accounting in the query profile is also corrected. `collect_race_result` and `_execute_sequential_peer_read` added the whole contiguous span size to `bytes_read_from_cross_cg_peer` / `bytes_read_from_same_cg_peer`, although peer returns only the requested ranges. On the sync path, DOWNLOADED blocks between the first and last EMPTY block were therefore counted as peer bytes (an existing overcount), and the sparse async read above would count its locally read middle blocks the same way. Both now add `PeerFetchResult::bytes_read`, the byte count `fetch_blocks()` verified, read before the result is moved out; the unused `span_size` parameter of `collect_race_result` is removed. The race lambda also captures `empty_ranges` and `candidates` by value directly instead of `std::move` on const references, which already copied. ### Release note None ### Check List (For Author) - Test: Unit Test - New CachedRemoteFileReaderAsyncPeerTest cases: miss-cached-miss with peer winning (peer gets two ranges, middle read locally, two writes submitted, second read local); the same span with peer failing (storage supplies only the misses, no double counting); an inflight middle block; a NOT_FOUND middle block rereads the span and self-heals; a downloading middle block and the whole-range fallback bypass peer. - New CrossCGAsyncWinnerRaceTest cases: sparse peer wins the race; an eligible span without candidates copies only the misses from storage. - Split and reordered peer responses: the mock peer service returns every requested block as uneven pieces, interleaved and out of order, for a sparse span and for an all-miss span, in attachment and protobuf-bytes modes. Each case checks the result bytes, the peer/local/remote byte counts, and the cache blocks read back directly after the async writes. With the chunk sort in take_peer_span removed for a local experiment (not committed), all four fail only on the cache read-back ("LIJK", "GHEF", "KLIJ" instead of the peer bytes) while the result bytes stay correct. - Peer byte accounting: sparse peer wins assert the CG byte counters equal the 8 bytes of the two missing blocks rather than the 12-byte span, for an async race win with a cross-CG candidate (async_race_sparse_peer_wins_around_cached_block), an async sequential read with a same-CG candidate (async_sequential_sparse_peer_counts_only_miss_bytes), and a sync EMPTY/DOWNLOADED/EMPTY race win (sync_peer_wins_counts_only_empty_block_bytes, which releases the S3 leg and waits for it to drop the reader before returning). All three read 12 before the fix. An all-miss race win still counts 12. - Before the change the new sparse cases fail (no peer RPC, whole span from storage, no self-heal for the middle block) and the counter cases fail to link. - CachedRemoteFileReaderPeerTest.*, CachedRemoteFileReaderAsyncPeerTest.*, CrossCGWinnerRaceTest.*, CrossCGAsyncWinnerRaceTest.*, AsyncCachedRemoteFileReaderTest.*, BlockFileCacheTest.*, AsyncCacheWriteManagerTest.*, InflightWriteBufferIndexTest.*, FileCacheProfileReporterTest.*: 286 tests, 283 pass and 3 pre-existing DISABLE_ tests are skipped (ASAN). The async peer and sync/async race cases (37 tests) pass 10 repeated runs. - Unit tests were run locally with GLIBC_COMPATIBILITY=OFF; there is no performance data yet. - Behavior changed: Yes. With async cache write and peer read enabled, a span whose misses surround cached or inflight blocks reads those blocks locally and requests only the misses from peer. - Does this need documentation: No

…hen misses surround local blocks ### What problem does this PR solve? Issue Number: N/A Related PR: apache#69071 Problem Summary: A peer fails the whole RPC when it misses any requested block. So on the async cache write read path, a remote span with cached or inflight blocks between its misses (miss-hit-miss) bypassed peer and was reread entirely from remote storage, while the sync path requests only its empty blocks from peer and reads the blocks in between locally. Align the async path with the sync path. When peer read applies, the tablet has a peer candidate, and the non-miss blocks inside the span are all CACHE or INFLIGHT: - Those blocks are read locally first with `_materialize_async_block` (CACHE blocks are touched in the LRU). Their bytes count as local only when all of them succeed. - Peer receives only the miss ranges. On a peer win the misses are copied into the result and their IOBufs are placed at the misses' span positions, so `_submit_async_write_tasks` is unchanged. - A storage fallback (peer failure, S3 winning the race, or the candidates disappearing after the check) still reads the whole span, from which only the misses are copied. The span keeps the existing whole-span storage read in three cases: it contains a DOWNLOADING block (the async path never waits on another downloader), it is the whole-range fallback after a side race (marked on the plan), or a local read inside the span fails; a NOT_FOUND there still triggers self-heal. When peer read does not apply, the path is unchanged. When the tablet has no usable candidate (none fetched yet, or in all-miss cooldown), reading the hits locally would only delay the whole-span storage read that follows, so the sparse read is skipped and the span is reread from storage as before this change. The check uses the new read-only `CloudWarmUpManager::has_peer_candidates()`, which, unlike `get_peer_candidates()`, does not refresh candidate expiry, reset an expired cooldown or touch bvars. Such a span also starts the background candidate fetch from FE (unless in cooldown), through the same helper `_execute_remote_read()` now uses; otherwise a tablet that only reads mixed spans would never fetch candidates. Candidates may change between the check and the read; either way the read falls back to storage, as before. Three bvars report how eligible spans use peer: - `cached_remote_file_reader_async_peer_eligible_span_count`: async-path remote spans that may use peer read (peer read enabled and not bypassed by the caller). - `cached_remote_file_reader_async_peer_no_candidate_span_count`: eligible spans with hits whose tablet has no usable candidate. This takes precedence over the span shape. - `cached_remote_file_reader_async_peer_bypass_span_count`: eligible spans with hits and candidates that read remote storage only because of one of the three shape cases above. Eligible spans made purely of misses, and sparse reads, are in neither. Spans that try peer but end on remote storage (peer miss or failure, S3 winning the race) are not counted either. Peer byte accounting in the query profile is also corrected. `collect_race_result` and `_execute_sequential_peer_read` added the whole contiguous span size to `bytes_read_from_cross_cg_peer` / `bytes_read_from_same_cg_peer`, although peer returns only the requested ranges. On the sync path, DOWNLOADED blocks between the first and last EMPTY block were therefore counted as peer bytes (an existing overcount), and the sparse async read above would count its locally read middle blocks the same way. Both now add `PeerFetchResult::bytes_read`, the byte count `fetch_blocks()` verified, read before the result is moved out; the unused `span_size` parameter of `collect_race_result` is removed. The race lambda also captures `empty_ranges` and `candidates` by value directly instead of `std::move` on const references, which already copied. The winner race gets two test sync points, `run_peer_race::peer_succeeded` and `CachedRemoteFileReader::_execute_winner_race::s3_published`, both hit after the leg releases `race->mtx`; the peer side is moved into a small `publish_peer_success()` helper. Race tests that expect a winner now wait for the other leg's signal instead of sleeping. The S3-win copy loop of the sparse read also checks the remote buffer bound. ### Release note None ### Check List (For Author) - Test: Unit Test - New CachedRemoteFileReaderAsyncPeerTest cases: miss-cached-miss with peer winning (peer gets two ranges, middle read locally, two writes submitted, second read local); the same span with peer failing (storage supplies only the misses, no double counting); an inflight middle block; a NOT_FOUND middle block rereads the span and self-heals; a downloading middle block and the whole-range fallback bypass peer. - New CrossCGAsyncWinnerRaceTest cases: sparse peer wins the race; an eligible mixed span without candidates skips the local reads (local bytes 0, remote 12), counts no_candidate and not bypass, and starts the candidate fetch (lazy-fetch counter +1, and the fetch is waited for); the same span in cooldown starts no fetch; a DOWNLOADING middle block without candidates counts no_candidate, not bypass. - CloudWarmUpManagerPeerTest.has_peer_candidates_is_read_only: unknown tablet, empty candidates and active cooldown return false; expired cooldown and present candidates return true; the candidate cache hit/miss bvars, last_access_time_ms and cooldown fields are unchanged. - The ten race tests that expect peer to win wait for peer_succeeded instead of an RPC count plus a 10 ms sleep, and race_peer_stops_when_s3_wins waits for s3_published instead of a 100 ms sleep. - Split and reordered peer responses: the mock peer service returns every requested block as uneven pieces, interleaved and out of order, for a sparse span and for an all-miss span, in attachment and protobuf-bytes modes. Each case checks the result bytes, the peer/local/remote byte counts, and the cache blocks read back directly after the async writes. With the chunk sort in take_peer_span removed for a local experiment (not committed), all four fail only on the cache read-back ("LIJK", "GHEF", "KLIJ" instead of the peer bytes) while the result bytes stay correct. - Peer byte accounting: sparse peer wins assert the CG byte counters equal the 8 bytes of the two missing blocks rather than the 12-byte span, for an async race win with a cross-CG candidate (async_race_sparse_peer_wins_around_cached_block), an async sequential read with a same-CG candidate (async_sequential_sparse_peer_counts_only_miss_bytes), and a sync EMPTY/DOWNLOADED/EMPTY race win (sync_peer_wins_counts_only_empty_block_bytes, which releases the S3 leg and waits for it to drop the reader before returning). All three read 12 before the fix. An all-miss race win still counts 12. - Before the no-candidate change the new no-candidate and has_peer_candidates tests fail to build. With only the candidate check in _prepare_async_sparse_peer removed for a local experiment (not committed), async_sparse_span_without_candidate_reads_storage_and_fetches fails on no_candidate (delta 0), bytes_read_from_local (4) and bytes_read_from_remote (8). Before the change the new sparse cases fail (no peer RPC, whole span from storage, no self-heal for the middle block) and the counter cases fail to link. - CachedRemoteFileReaderPeerTest.*, CachedRemoteFileReaderAsyncPeerTest.*, CrossCGWinnerRaceTest.*, CrossCGAsyncWinnerRaceTest.*, AsyncCachedRemoteFileReaderTest.*, BlockFileCacheTest.*, AsyncCacheWriteManagerTest.*, InflightWriteBufferIndexTest.*, FileCacheProfileReporterTest.*, CloudWarmUpManagerPeerTest.*: 318 tests, 315 pass and 3 pre-existing DISABLE_ tests are skipped (ASAN). The async peer and race cases (38 tests) pass 20 repeated runs. - Unit tests were run locally with GLIBC_COMPATIBILITY=OFF; there is no performance data yet. - Behavior changed: Yes. With async cache write and peer read enabled, a span whose misses surround cached or inflight blocks reads those blocks locally and requests only the misses from peer. - Does this need documentation: No

…hen misses surround local blocks ### What problem does this PR solve? Issue Number: N/A Related PR: apache#69071 Problem Summary: A peer fails the whole RPC when it misses any requested block. So on the async cache write read path, a remote span with cached or inflight blocks between its misses (miss-hit-miss) bypassed peer and was reread entirely from remote storage, while the sync path requests only its empty blocks from peer and reads the blocks in between locally. Align the async path with the sync path. When peer read applies, the tablet has a peer candidate, and the non-miss blocks inside the span are all CACHE or INFLIGHT: - Those blocks are read locally first with `_materialize_async_block` (CACHE blocks are touched in the LRU). Their bytes count as local only when all of them succeed. - Peer receives only the miss ranges. On a peer win the misses are copied into the result and their IOBufs are placed at the misses' span positions, so `_submit_async_write_tasks` is unchanged. - A storage fallback (peer failure, S3 winning the race, or the candidates disappearing after the check) still reads the whole span, from which only the misses are copied. The span keeps the existing whole-span storage read in three cases: it contains a DOWNLOADING block (the async path never waits on another downloader), it is the whole-range fallback after a side race (marked on the plan), or a local read inside the span fails; a NOT_FOUND there still triggers self-heal. When peer read does not apply, the path is unchanged. When the tablet has no usable candidate (none fetched yet, or in all-miss cooldown), reading the hits locally would only delay the whole-span storage read that follows, so the sparse read is skipped and the span is reread from storage as before this change. The check uses the new read-only `CloudWarmUpManager::has_peer_candidates()`, which, unlike `get_peer_candidates()`, does not refresh candidate expiry, reset an expired cooldown or touch bvars. Such a span also starts the background candidate fetch from FE (unless in cooldown), through the same helper `_execute_remote_read()` now uses; otherwise a tablet that only reads mixed spans would never fetch candidates. Candidates may change between the check and the read; either way the read falls back to storage, as before. Three bvars report how eligible spans use peer: - `cached_remote_file_reader_async_peer_eligible_span_count`: async-path remote spans that may use peer read (peer read enabled and not bypassed by the caller). - `cached_remote_file_reader_async_peer_no_candidate_span_count`: eligible spans with hits whose tablet has no usable candidate. This takes precedence over the span shape. - `cached_remote_file_reader_async_peer_bypass_span_count`: eligible spans with hits and candidates that read remote storage only because of one of the three shape cases above. Eligible spans made purely of misses, and sparse reads, are in neither. Spans that try peer but end on remote storage (peer miss or failure, S3 winning the race) are not counted either. Peer byte accounting in the query profile is also corrected. `collect_race_result` and `_execute_sequential_peer_read` added the whole contiguous span size to `bytes_read_from_cross_cg_peer` / `bytes_read_from_same_cg_peer`, although peer returns only the requested ranges. On the sync path, DOWNLOADED blocks between the first and last EMPTY block were therefore counted as peer bytes (an existing overcount), and the sparse async read above would count its locally read middle blocks the same way. Both now add `PeerFetchResult::bytes_read`, the byte count `fetch_blocks()` verified, read before the result is moved out; the unused `span_size` parameter of `collect_race_result` is removed. The race lambda now init-captures `empty_ranges` and `candidates`, copying each once into a non-const member that the `mutable` lambda moves into `run_peer_race`. Before, it captured the const reference parameters by value, so the members were const and the `std::move` in the lambda copied a second time. The winner race gets two test sync points, `run_peer_race::peer_succeeded` and `CachedRemoteFileReader::_execute_winner_race::s3_published`, both hit after the leg releases `race->mtx`; the peer side is moved into a small `publish_peer_success()` helper. Race tests that expect a winner now wait for the other leg's signal instead of sleeping. The S3-win copy loop of the sparse read also checks the remote buffer bound. PeerRaceS3ThreadPool runs the S3 leg of every peer/S3 winner race, on the sync and async paths alike. Its size, max_peer_race_s3_thread_num, defaults to 32, smaller than the 64 race slots of max_concurrent_peer_races, and its queue is unbounded. With more running races than threads, S3 legs queue behind each other, and raising the race slots alone makes it worse. On a two-BE cluster (both PL1 disks, SSB SF1000 lineorder full scan, async write + peer; peer-async-perf-exp1-results.md): - 64 scanners, where the race slots are not the limit (0 serial peer reads, race peak 63): 64 slots / 32 threads (E0/64/C/1..3) took 169.20 s with an S3 pool queue peak of 16; 256 slots / 256 threads (E1b/64/C/1..3) took 144.28 s (-14.7%) with a queue peak of 0. - 128 scanners: 64 slots / 32 threads (E0/128/C/1..3) took 124.24 s; 256 slots with 32 threads (E1a/128/C/1..3) took 172.51 s (+38.9%) with the queue at 103-264 and about 110 ms wait per S3 task; 256 slots / 256 threads (E1b/128/C/1..3) took 77.42 s (-55.1% vs E1a). The pool is now created with max(max_peer_race_s3_thread_num, max_concurrent_peer_races) threads, so every race slot has an S3 thread, and BE logs the effective size with both config values. Changing max_concurrent_peer_races at runtime does not resize the pool; the config comment says so. ### Release note None ### Check List (For Author) - Test: Unit Test - New CachedRemoteFileReaderAsyncPeerTest cases: miss-cached-miss with peer winning (peer gets two ranges, middle read locally, two writes submitted, second read local); the same span with peer failing (storage supplies only the misses, no double counting); an inflight middle block; a NOT_FOUND middle block rereads the span and self-heals; a downloading middle block and the whole-range fallback bypass peer. - New CrossCGAsyncWinnerRaceTest cases: sparse peer wins the race; an eligible mixed span without candidates skips the local reads (local bytes 0, remote 12), counts no_candidate and not bypass, and starts the candidate fetch (lazy-fetch counter +1, and the fetch is waited for); the same span in cooldown starts no fetch; a DOWNLOADING middle block without candidates counts no_candidate, not bypass. - CloudWarmUpManagerPeerTest.has_peer_candidates_*: four tests (unknown or empty tablet, cooling down, cooldown expired, with candidates). Unknown tablet, empty candidates and active cooldown return false; expired cooldown and present candidates return true; after each call the candidate cache hit/miss bvars, last_access_time_ms and cooldown fields are unchanged. - The ten race tests that expect peer to win wait for peer_succeeded instead of an RPC count plus a 10 ms sleep, and race_peer_stops_when_s3_wins waits for s3_published instead of a 100 ms sleep. - Split and reordered peer responses: the mock peer service returns every requested block as uneven pieces, interleaved and out of order, for a sparse span and for an all-miss span, in attachment and protobuf-bytes modes. Each case checks the result bytes, the peer/local/remote byte counts, and the cache blocks read back directly after the async writes. With the chunk sort in take_peer_span removed for a local experiment (not committed), all four fail only on the cache read-back ("LIJK", "GHEF", "KLIJ" instead of the peer bytes) while the result bytes stay correct. - Peer byte accounting: sparse peer wins assert the CG byte counters equal the 8 bytes of the two missing blocks rather than the 12-byte span, for an async race win with a cross-CG candidate (async_race_sparse_peer_wins_around_cached_block), an async sequential read with a same-CG candidate (async_sequential_sparse_peer_counts_only_miss_bytes), and a sync EMPTY/DOWNLOADED/EMPTY race win (sync_peer_wins_counts_only_empty_block_bytes, which releases the S3 leg and waits for it to drop the reader before returning). All three read 12 before the fix. An all-miss race win still counts 12. - Before the no-candidate change the new no-candidate and has_peer_candidates tests fail to build. With only the candidate check in _prepare_async_sparse_peer removed for a local experiment (not committed), async_sparse_span_without_candidate_reads_storage_and_fetches fails on no_candidate (delta 0), bytes_read_from_local (4) and bytes_read_from_remote (8). Before the change the new sparse cases fail (no peer RPC, whole span from storage, no self-heal for the middle block) and the counter cases fail to link. - CachedRemoteFileReaderPeerTest.*, CachedRemoteFileReaderAsyncPeerTest.*, CrossCGWinnerRaceTest.*, CrossCGAsyncWinnerRaceTest.*, AsyncCachedRemoteFileReaderTest.*, BlockFileCacheTest.*, AsyncCacheWriteManagerTest.*, InflightWriteBufferIndexTest.*, FileCacheProfileReporterTest.*, CloudWarmUpManagerPeerTest.*: 318 tests, 315 pass and 3 pre-existing DISABLE_ tests are skipped (ASAN). The async peer and race cases (38 tests) pass 20 repeated runs. - Regression test (docker, cloud): cloud_p0/cache/read_from_peer/test_async_file_cache_write_peer runs two compute groups with async cache writes and the peer/S3 race. It warms CG-A until a query reads nothing from storage, lets CG-B fetch candidates, then checks on CG-B: an empty cache reads its misses from CG-A through peer (peer success, peer race wins, async submissions and eligible spans increase, no no_candidate span); after the async writes finish the same query reads neither peer nor storage; a partially warmed cache returns the same result; with peer read disabled at runtime nothing is read from peer. Every result matches CG-A. Passed on three consecutive runs (one earlier run passed every assertion but its cluster teardown exceeded the framework's 60 s compose timeout). With peer read disabled on CG-B just before the peer step (local experiment, not committed) it fails on the peer success, peer win and eligible span deltas, all 0. - Fixed an empty wait in test_async_file_cache_write: it polled async_cache_write_pending_task_count, which does not exist in be/src, so the wait never waited. It now waits for async_cache_write_buffer_memory_bytes to return to 0. The .out is unchanged and the case passed three runs. - Unit tests were run locally with GLIBC_COMPATIBILITY=OFF. - Performance: tested on a real cluster (see the Performance section of the PR description). With the default race config, fast peer source and PL1 reader: async+peer vs async+S3 −56.6% (64 scanners) / −20.0% (128), S3 GETs about −90%. - Behavior changed: Yes. With async cache write and peer read enabled, a span whose misses surround cached or inflight blocks reads those blocks locally and requests only the misses from peer. The default effective size of PeerRaceS3ThreadPool grows from 32 to 64 threads (equal to max_concurrent_peer_races), which also affects the peer/S3 race on the sync path. - Does this need documentation: No

deardeng · 2026-10-26T03:51:27Z

run buildall

shuke987 · 2026-10-26T04:39:22Z

run buildall

hello-stephen · 2026-10-26T08:19:56Z

TPC-H: Total hot run time: 28461 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8895b698d060c98308e8184de59108afadf49806, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17201	5079	5032	5032
q2	2111	343	247	247
q3	9999	1385	818	818
q4	4671	471	334	334
q5	7436	811	539	539
q6	178	186	151	151
q7	733	807	610	610
q8	9252	1341	1150	1150
q9	5484	4210	4201	4201
q10	6823	1313	1024	1024
q11	431	255	229	229
q12	647	413	278	278
q13	18119	3649	3000	3000
q14	274	275	243	243
q15	q16	727	724	667	667
q17	1213	745	578	578
q18	7007	6172	6127	6127
q19	1251	969	602	602
q20	394	338	231	231
q21	5032	2145	2250	2145
q22	353	299	255	255
Total cold run time: 99336 ms
Total hot run time: 28461 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5604	5560	5531	5531
q2	671	589	525	525
q3	5227	5410	4924	4924
q4	1975	2137	1521	1521
q5	5307	5202	5536	5202
q6	266	212	154	154
q7	2373	2079	1912	1912
q8	3048	2862	2622	2622
q9	8120	7980	7988	7980
q10	3733	3678	3260	3260
q11	629	442	417	417
q12	680	678	511	511
q13	3484	3980	3247	3247
q14	323	313	305	305
q15	q16	699	706	638	638
q17	8286	7506	7264	7264
q18	13117	12277	12877	12277
q19	898	780	771	771
q20	2154	2127	1941	1941
q21	5635	4548	4686	4548
q22	498	449	412	412
Total cold run time: 72727 ms
Total hot run time: 65962 ms

hello-stephen · 2026-10-26T08:29:40Z

TPC-DS: Total hot run time: 150482 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8895b698d060c98308e8184de59108afadf49806, data reload: false

query5	4315	591	438	438
query6	424	205	191	191
query7	4796	455	220	220
query8	315	165	148	148
query9	8773	3125	3103	3103
query10	484	315	248	248
query11	6056	3833	3576	3576
query12	145	92	86	86
query13	1242	435	314	314
query14	6114	3934	3643	3643
query14_1	3452	3377	3421	3377
query15	209	190	180	180
query16	926	413	408	408
query17	845	659	579	579
query18	2402	408	295	295
query19	202	159	128	128
query20	81	81	84	81
query21	207	124	101	101
query22	13049	13067	12768	12768
query23	13451	12841	12417	12417
query23_1	12677	12617	12623	12617
query24	7248	815	451	451
query24_1	454	460	452	452
query25	504	399	332	332
query26	1251	255	133	133
query27	2768	438	266	266
query28	4554	1934	1944	1934
query29	1508	568	430	430
query30	307	221	191	191
query31	1011	936	830	830
query32	140	92	87	87
query33	505	289	231	231
query34	892	851	503	503
query35	737	746	640	640
query36	792	816	729	729
query37	126	97	82	82
query38	1774	1728	1658	1658
query39	784	778	746	746
query39_1	740	726	733	726
query40	224	132	93	93
query41	65	63	64	63
query42	84	79	114	79
query43	347	356	314	314
query44	1304	683	683	683
query45	177	178	159	159
query46	819	958	564	564
query47	2804	2904	2867	2867
query48	294	306	213	213
query49	567	393	293	293
query50	685	270	207	207
query51	10253	10504	10128	10128
query52	76	79	69	69
query53	187	205	154	154
query54	237	189	179	179
query55	69	64	67	64
query56	243	207	198	198
query57	1636	1607	1560	1560
query58	282	261	256	256
query59	2129	2133	1952	1952
query60	274	240	232	232
query61	145	151	140	140
query62	394	332	292	292
query63	186	156	155	155
query64	2672	909	794	794
query65	3319	3283	3271	3271
query66	1834	409	304	304
query67	20428	20152	19958	19958
query68	3049	1004	611	611
query69	381	301	246	246
query70	853	770	756	756
query71	293	238	218	218
query72	2702	2606	2383	2383
query73	511	514	292	292
query74	4637	4503	4353	4353
query75	2341	2312	1915	1915
query76	2234	1016	602	602
query77	360	376	267	267
query78	9573	9668	9167	9167
query79	875	823	517	517
query80	484	400	337	337
query81	520	310	272	272
query82	199	130	108	108
query83	200	199	182	182
query84	290	113	96	96
query85	749	412	370	370
query86	256	237	224	224
query87	1964	1927	1839	1839
query88	3555	2675	2643	2643
query89	312	283	232	232
query90	2029	168	166	166
query91	151	145	125	125
query92	98	88	90	88
query93	1003	993	571	571
query94	451	313	265	265
query95	581	337	364	337
query96	659	490	237	237
query97	2411	2421	2317	2317
query98	159	164	147	147
query99	753	765	658	658
Total cold run time: 232425 ms
Total hot run time: 150482 ms

hello-stephen · 2026-10-26T08:34:31Z

ClickBench: Total hot run time: 24.35 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8895b698d060c98308e8184de59108afadf49806, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.25	0.08	0.08
query4	1.63	0.10	0.09
query5	0.24	0.24	0.25
query6	1.17	0.69	0.68
query7	0.04	0.01	0.01
query8	0.05	0.03	0.03
query9	0.40	0.35	0.36
query10	0.56	0.59	0.59
query11	0.20	0.15	0.14
query12	0.18	0.15	0.14
query13	0.50	0.49	0.51
query14	1.03	1.03	1.01
query15	0.62	0.61	0.60
query16	0.36	0.36	0.35
query17	1.11	1.14	1.10
query18	0.22	0.20	0.19
query19	2.10	2.06	1.91
query20	0.02	0.02	0.01
query21	15.48	0.19	0.13
query22	4.86	0.05	0.05
query23	16.28	0.33	0.13
query24	3.04	0.49	0.48
query25	0.10	0.05	0.07
query26	1.27	0.23	0.15
query27	0.05	0.03	0.05
query28	3.18	0.79	0.36
query29	12.49	4.19	3.26
query30	0.28	0.15	0.10
query31	2.77	0.54	0.33
query32	3.22	0.60	0.51
query33	3.14	3.17	3.16
query34	15.43	4.02	3.42
query35	3.36	3.34	3.34
query36	0.55	0.49	0.42
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.03	0.03
query40	0.16	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 96.77 s
Total hot run time: 24.35 s

hello-stephen · 2026-10-26T08:53:32Z

BE Regression && UT Coverage Report

Increment line coverage 60.00% (6/10) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.33% (36037/46600)
Line Coverage	62.56% (407186/650895)
Region Coverage	59.00% (343903/582870)
Branch Coverage	59.95% (159227/265581)

…hen misses surround local blocks Issue Number: N/A Related PR: apache#69071 Problem Summary: A peer fails the whole RPC when it misses any requested block. So on the async cache write read path, a remote span with cached or inflight blocks between its misses (miss-hit-miss) bypassed peer and was reread entirely from remote storage, while the sync path requests only its empty blocks from peer and reads the blocks in between locally. Align the async path with the sync path. When peer read applies, the tablet has a peer candidate, and the non-miss blocks inside the span are all CACHE or INFLIGHT: - Those blocks are read locally first with `_materialize_async_block` (CACHE blocks are touched in the LRU). Their bytes count as local only when all of them succeed. - Peer receives only the miss ranges. On a peer win the misses are copied into the result and their IOBufs are placed at the misses' span positions, so `_submit_async_write_tasks` is unchanged. - A storage fallback (peer failure, S3 winning the race, or the candidates disappearing after the check) still reads the whole span, from which only the misses are copied. The span keeps the existing whole-span storage read in three cases: it contains a DOWNLOADING block (the async path never waits on another downloader), it is the whole-range fallback after a side race (marked on the plan), or a local read inside the span fails; a NOT_FOUND there still triggers self-heal. When peer read does not apply, the path is unchanged. When the tablet has no usable candidate (none fetched yet, or in all-miss cooldown), reading the hits locally would only delay the whole-span storage read that follows, so the sparse read is skipped and the span is reread from storage as before this change. The check uses the new read-only `CloudWarmUpManager::has_peer_candidates()`, which, unlike `get_peer_candidates()`, does not refresh candidate expiry, reset an expired cooldown or touch bvars. Such a span also starts the background candidate fetch from FE (unless in cooldown), through the same helper `_execute_remote_read()` now uses; otherwise a tablet that only reads mixed spans would never fetch candidates. Candidates may change between the check and the read; either way the read falls back to storage, as before. Three bvars report how eligible spans use peer: - `cached_remote_file_reader_async_peer_eligible_span_count`: async-path remote spans that may use peer read (peer read enabled and not bypassed by the caller). - `cached_remote_file_reader_async_peer_no_candidate_span_count`: eligible spans with hits whose tablet has no usable candidate. This takes precedence over the span shape. - `cached_remote_file_reader_async_peer_bypass_span_count`: eligible spans with hits and candidates that read remote storage only because of one of the three shape cases above. Eligible spans made purely of misses, and sparse reads, are in neither. Spans that try peer but end on remote storage (peer miss or failure, S3 winning the race) are not counted either. Peer byte accounting in the query profile is also corrected. `collect_race_result` and `_execute_sequential_peer_read` added the whole contiguous span size to `bytes_read_from_cross_cg_peer` / `bytes_read_from_same_cg_peer`, although peer returns only the requested ranges. On the sync path, DOWNLOADED blocks between the first and last EMPTY block were therefore counted as peer bytes (an existing overcount), and the sparse async read above would count its locally read middle blocks the same way. Both now add `PeerFetchResult::bytes_read`, the byte count `fetch_blocks()` verified, read before the result is moved out; the unused `span_size` parameter of `collect_race_result` is removed. The race lambda now init-captures `empty_ranges` and `candidates`, copying each once into a non-const member that the `mutable` lambda moves into `run_peer_race`. Before, it captured the const reference parameters by value, so the members were const and the `std::move` in the lambda copied a second time. The winner race gets two test sync points, `run_peer_race::peer_succeeded` and `CachedRemoteFileReader::_execute_winner_race::s3_published`, both hit after the leg releases `race->mtx`; the peer side is moved into a small `publish_peer_success()` helper. Race tests that expect a winner now wait for the other leg's signal instead of sleeping. The S3-win copy loop of the sparse read also checks the remote buffer bound. PeerRaceS3ThreadPool runs the S3 leg of every peer/S3 winner race, on the sync and async paths alike. Its size, max_peer_race_s3_thread_num, defaults to 32, smaller than the 64 race slots of max_concurrent_peer_races, and its queue is unbounded. With more running races than threads, S3 legs queue behind each other, and raising the race slots alone makes it worse. On a two-BE cluster (both PL1 disks, SSB SF1000 lineorder full scan, async write + peer; peer-async-perf-exp1-results.md): - 64 scanners, where the race slots are not the limit (0 serial peer reads, race peak 63): 64 slots / 32 threads (E0/64/C/1..3) took 169.20 s with an S3 pool queue peak of 16; 256 slots / 256 threads (E1b/64/C/1..3) took 144.28 s (-14.7%) with a queue peak of 0. - 128 scanners: 64 slots / 32 threads (E0/128/C/1..3) took 124.24 s; 256 slots with 32 threads (E1a/128/C/1..3) took 172.51 s (+38.9%) with the queue at 103-264 and about 110 ms wait per S3 task; 256 slots / 256 threads (E1b/128/C/1..3) took 77.42 s (-55.1% vs E1a). The pool is now created with max(max_peer_race_s3_thread_num, max_concurrent_peer_races) threads, so every race slot has an S3 thread, and BE logs the effective size with both config values. Changing max_concurrent_peer_races at runtime does not resize the pool; the config comment says so. None - Test: Unit Test - New CachedRemoteFileReaderAsyncPeerTest cases: miss-cached-miss with peer winning (peer gets two ranges, middle read locally, two writes submitted, second read local); the same span with peer failing (storage supplies only the misses, no double counting); an inflight middle block; a NOT_FOUND middle block rereads the span and self-heals; a downloading middle block and the whole-range fallback bypass peer. - New CrossCGAsyncWinnerRaceTest cases: sparse peer wins the race; an eligible mixed span without candidates skips the local reads (local bytes 0, remote 12), counts no_candidate and not bypass, and starts the candidate fetch (lazy-fetch counter +1, and the fetch is waited for); the same span in cooldown starts no fetch; a DOWNLOADING middle block without candidates counts no_candidate, not bypass. - CloudWarmUpManagerPeerTest.has_peer_candidates_*: four tests (unknown or empty tablet, cooling down, cooldown expired, with candidates). Unknown tablet, empty candidates and active cooldown return false; expired cooldown and present candidates return true; after each call the candidate cache hit/miss bvars, last_access_time_ms and cooldown fields are unchanged. - The ten race tests that expect peer to win wait for peer_succeeded instead of an RPC count plus a 10 ms sleep, and race_peer_stops_when_s3_wins waits for s3_published instead of a 100 ms sleep. - Split and reordered peer responses: the mock peer service returns every requested block as uneven pieces, interleaved and out of order, for a sparse span and for an all-miss span, in attachment and protobuf-bytes modes. Each case checks the result bytes, the peer/local/remote byte counts, and the cache blocks read back directly after the async writes. With the chunk sort in take_peer_span removed for a local experiment (not committed), all four fail only on the cache read-back ("LIJK", "GHEF", "KLIJ" instead of the peer bytes) while the result bytes stay correct. - Peer byte accounting: sparse peer wins assert the CG byte counters equal the 8 bytes of the two missing blocks rather than the 12-byte span, for an async race win with a cross-CG candidate (async_race_sparse_peer_wins_around_cached_block), an async sequential read with a same-CG candidate (async_sequential_sparse_peer_counts_only_miss_bytes), and a sync EMPTY/DOWNLOADED/EMPTY race win (sync_peer_wins_counts_only_empty_block_bytes, which releases the S3 leg and waits for it to drop the reader before returning). All three read 12 before the fix. An all-miss race win still counts 12. - Before the no-candidate change the new no-candidate and has_peer_candidates tests fail to build. With only the candidate check in _prepare_async_sparse_peer removed for a local experiment (not committed), async_sparse_span_without_candidate_reads_storage_and_fetches fails on no_candidate (delta 0), bytes_read_from_local (4) and bytes_read_from_remote (8). Before the change the new sparse cases fail (no peer RPC, whole span from storage, no self-heal for the middle block) and the counter cases fail to link. - CachedRemoteFileReaderPeerTest.*, CachedRemoteFileReaderAsyncPeerTest.*, CrossCGWinnerRaceTest.*, CrossCGAsyncWinnerRaceTest.*, AsyncCachedRemoteFileReaderTest.*, BlockFileCacheTest.*, AsyncCacheWriteManagerTest.*, InflightWriteBufferIndexTest.*, FileCacheProfileReporterTest.*, CloudWarmUpManagerPeerTest.*: 318 tests, 315 pass and 3 pre-existing DISABLE_ tests are skipped (ASAN). The async peer and race cases (38 tests) pass 20 repeated runs. - Regression test (docker, cloud): cloud_p0/cache/read_from_peer/test_async_file_cache_write_peer runs two compute groups with async cache writes and the peer/S3 race. It warms CG-A until a query reads nothing from storage, lets CG-B fetch candidates, then checks on CG-B: an empty cache reads its misses from CG-A through peer (peer success, peer race wins, async submissions and eligible spans increase, no no_candidate span); after the async writes finish the same query reads neither peer nor storage; a partially warmed cache returns the same result; with peer read disabled at runtime nothing is read from peer. Every result matches CG-A. Passed on three consecutive runs (one earlier run passed every assertion but its cluster teardown exceeded the framework's 60 s compose timeout). With peer read disabled on CG-B just before the peer step (local experiment, not committed) it fails on the peer success, peer win and eligible span deltas, all 0. - Fixed an empty wait in test_async_file_cache_write: it polled async_cache_write_pending_task_count, which does not exist in be/src, so the wait never waited. It now waits for async_cache_write_buffer_memory_bytes to return to 0. The .out is unchanged and the case passed three runs. - Unit tests were run locally with GLIBC_COMPATIBILITY=OFF. - Performance: tested on a real cluster (see the Performance section of the PR description). With the default race config, fast peer source and PL1 reader: async+peer vs async+S3 −56.6% (64 scanners) / −20.0% (128), S3 GETs about −90%. - Behavior changed: Yes. With async cache write and peer read enabled, a span whose misses surround cached or inflight blocks reads those blocks locally and requests only the misses from peer. The default effective size of PeerRaceS3ThreadPool grows from 32 to 64 threads (equal to max_concurrent_peer_races), which also affects the peer/S3 race on the sync path. - Does this need documentation: No

deardeng · 2026-10-29T10:30:45Z

run buildall

hello-stephen · 2026-10-29T13:55:19Z

TPC-H: Total hot run time: 28783 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aa73a107121acd4e8dd6a64a242acd8c8663488f, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17573	5139	5030	5030
q2	2143	313	297	297
q3	10187	1435	803	803
q4	4686	463	334	334
q5	7423	820	524	524
q6	178	191	156	156
q7	748	803	590	590
q8	9243	1357	1086	1086
q9	5630	4230	4222	4222
q10	6835	1278	1041	1041
q11	448	245	224	224
q12	637	396	276	276
q13	18165	3581	3010	3010
q14	277	276	247	247
q15	q16	732	714	644	644
q17	1234	785	591	591
q18	7027	6173	6108	6108
q19	1114	971	625	625
q20	385	338	224	224
q21	5908	2547	2485	2485
q22	396	331	266	266
Total cold run time: 100969 ms
Total hot run time: 28783 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5933	5903	6115	5903
q2	720	607	599	599
q3	5892	5786	5347	5347
q4	2024	2179	1550	1550
q5	5757	5757	5631	5631
q6	239	186	145	145
q7	2276	2082	1814	1814
q8	3033	2622	2723	2622
q9	8019	7660	7506	7506
q10	3640	3542	3108	3108
q11	621	399	389	389
q12	680	659	519	519
q13	3289	3606	3056	3056
q14	298	297	258	258
q15	q16	653	670	610	610
q17	7774	6991	6858	6858
q18	12948	12205	12964	12205
q19	909	796	787	787
q20	2124	2125	1911	1911
q21	5613	4617	4714	4617
q22	486	435	393	393
Total cold run time: 72928 ms
Total hot run time: 65828 ms

hello-stephen · 2026-10-29T14:05:10Z

TPC-DS: Total hot run time: 151329 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aa73a107121acd4e8dd6a64a242acd8c8663488f, data reload: false

query5	4313	581	440	440
query6	421	219	189	189
query7	4811	448	223	223
query8	318	166	156	156
query9	8802	3212	3218	3212
query10	473	327	264	264
query11	6030	3850	3575	3575
query12	153	90	85	85
query13	1248	449	330	330
query14	6119	3897	3628	3628
query14_1	3423	3386	3440	3386
query15	212	205	185	185
query16	973	454	336	336
query17	849	683	545	545
query18	2425	419	306	306
query19	205	159	139	139
query20	85	84	80	80
query21	210	127	105	105
query22	12902	12994	12906	12906
query23	13513	12957	12455	12455
query23_1	12397	12591	12647	12591
query24	7376	829	478	478
query24_1	493	481	495	481
query25	535	436	355	355
query26	1262	257	142	142
query27	2772	440	278	278
query28	4619	2055	2032	2032
query29	1574	612	468	468
query30	296	223	181	181
query31	1015	930	826	826
query32	161	96	90	90
query33	528	301	241	241
query34	899	855	505	505
query35	739	752	647	647
query36	832	813	766	766
query37	119	95	86	86
query38	1763	1766	1674	1674
query39	787	787	749	749
query39_1	738	732	733	732
query40	220	119	100	100
query41	74	70	69	69
query42	89	88	83	83
query43	343	367	312	312
query44	1302	706	737	706
query45	180	184	167	167
query46	837	970	558	558
query47	2891	2948	2826	2826
query48	302	312	216	216
query49	565	406	291	291
query50	671	270	199	199
query51	10369	10442	10298	10298
query52	88	82	66	66
query53	190	207	151	151
query54	228	198	176	176
query55	71	70	64	64
query56	255	220	218	218
query57	1644	1634	1559	1559
query58	293	274	250	250
query59	2193	2203	2036	2036
query60	288	238	223	223
query61	150	149	143	143
query62	390	336	301	301
query63	192	161	163	161
query64	2791	923	774	774
query65	3350	3273	3265	3265
query66	1784	409	309	309
query67	20349	20143	19944	19944
query68	3252	1021	626	626
query69	391	300	260	260
query70	843	774	752	752
query71	298	260	220	220
query72	2867	2525	2314	2314
query73	526	521	292	292
query74	4608	4505	4343	4343
query75	2352	2278	1951	1951
query76	2410	1019	596	596
query77	362	381	278	278
query78	9870	9706	9146	9146
query79	976	914	541	541
query80	817	424	361	361
query81	592	327	274	274
query82	790	129	99	99
query83	299	200	178	178
query84	318	127	100	100
query85	903	426	371	371
query86	350	235	248	235
query87	2000	1915	1786	1786
query88	3753	2748	2733	2733
query89	335	282	249	249
query90	1796	166	196	166
query91	167	156	131	131
query92	98	87	81	81
query93	998	1001	577	577
query94	524	300	274	274
query95	582	386	310	310
query96	684	485	238	238
query97	2463	2441	2343	2343
query98	164	150	150	150
query99	762	760	642	642
Total cold run time: 235429 ms
Total hot run time: 151329 ms

hello-stephen · 2026-10-29T14:10:02Z

ClickBench: Total hot run time: 24.13 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aa73a107121acd4e8dd6a64a242acd8c8663488f, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.26	0.09	0.08
query4	1.60	0.10	0.11
query5	0.23	0.25	0.24
query6	1.17	0.67	0.67
query7	0.04	0.00	0.01
query8	0.06	0.04	0.04
query9	0.38	0.35	0.36
query10	0.58	0.56	0.58
query11	0.20	0.15	0.14
query12	0.18	0.16	0.14
query13	0.49	0.49	0.48
query14	1.02	1.00	0.99
query15	0.62	0.61	0.60
query16	0.36	0.34	0.35
query17	1.11	1.17	1.16
query18	0.20	0.19	0.18
query19	2.03	1.96	1.97
query20	0.02	0.01	0.02
query21	15.46	0.19	0.14
query22	5.02	0.05	0.05
query23	16.22	0.31	0.13
query24	3.08	0.51	0.34
query25	0.13	0.07	0.04
query26	1.22	0.21	0.15
query27	0.05	0.05	0.06
query28	3.57	0.82	0.36
query29	12.51	4.13	3.24
query30	0.28	0.13	0.10
query31	2.77	0.55	0.33
query32	3.23	0.60	0.51
query33	3.27	3.13	3.20
query34	15.43	4.00	3.39
query35	3.35	3.31	3.35
query36	0.55	0.50	0.42
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.14
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 97.31 s
Total hot run time: 24.13 s

hello-stephen · 2026-10-29T14:42:19Z

BE Regression && UT Coverage Report

Increment line coverage 60.00% (6/10) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.58% (36421/46944)
Line Coverage	62.84% (411700/655193)
Region Coverage	59.29% (348581/587952)
Branch Coverage	60.19% (161594/268486)

…te path Related PR: apache#65658 Problem Summary: The async file cache write path added in apache#65658 did not support peer reads. `_resolve_cache_write_mode` forced SYNC_WRITE whenever peer read was enabled, and `_read_async_write_path` set `bypass_peer_read` on a copy of the IOContext. In a cloud deployment with peer read enabled, a cache miss therefore always took the sync path, writing and finalizing cache blocks on the query thread. The existing peer chain is zero-copy: the server reads cache files into an IOBuf, the client cuts the attachment, and the sync path writes it with `append_iobuf` (writev). Routing peer data through the async path must not add a flattening copy. 1. Peer chain takes block ranges `PeerFileCacheReader::fetch_blocks` and the peer dispatch helpers (`execute_peer_read`, `run_peer_race`, `_execute_sequential_peer_read`, `_execute_remote_read`, `_execute_winner_race`) took `std::vector<FileBlockSPtr>` but only read `block->range()`. The async path has no `FileBlock` for its misses, so the chain now takes `std::vector<FileBlock::Range>`. The sync caller converts its empty blocks before dispatch. A peer bthread that loses the race no longer holds `FileBlockSPtr`s. 2. Async write buffers can hold an IOBuf `AsyncCacheWriteManager::wrap_iobuf(capacity, payload)` wraps one cache block of peer payload without copying. It reports `capacity` as `size()`, so pending-byte accounting, the fixed task buffer size check and EOF short-block detection are unchanged. brpc allocates IOBuf blocks with malloc, outside any Doris MemTracker, so the wrapper charges the payload length to the async-write tracker on construction and releases it on destruction. `copy_to()` serves inflight readers and `append_to()` serves worker persistence through `FileBlock::append_iobuf`. Allocated buffers behave as before. 3. Async path reads from peers - Mode resolution no longer depends on peer read, and the async path no longer forces `bypass_peer_read`. - A peer fails the whole RPC when it misses any requested block, so peer is only tried when every block in the remote span is a real cache miss. A span that rereads local hits (miss-hit-miss, or the whole-range fallback after a side race) goes straight to remote storage. - On a peer win, chunks are copied to the user result from their IOBufs, regrouped by reference into one IOBuf per span block and wrapped as task buffers. The inflight index and the write worker use the received IOBuf blocks directly; the copy into the user buffer is the only copy. Storage results keep the existing tracked-buffer copy. - Peer bytes copied to the user are accounted as `peer_bytes`. 4. Profile timers Remote IO is already reported as `PeerIOUseTimer` / `RemoteIOUseTimer`. Add `AsyncCopyToUserTimer` (copying the remote/peer span into the user buffer) and `AsyncSubmitWriteTimer` (`_submit_async_write_tasks`) to the file cache profile, so a peer-miss read on the async path can be broken down and compared with the sync path's `WriteCacheIOUseTimer`. Both are included in the file cache group total. No metric or bvar is added. The async file cache write path (`enable_async_file_cache_write`) now reads missing blocks from peer BEs when `enable_cache_read_from_peer` is enabled, instead of falling back to synchronous cache writes. Add AsyncCopyToUserTimer and AsyncSubmitWriteTimer to the file cache profile. - Test: Unit Test - AsyncCacheWriteManagerTest: a multi-segment IOBuf payload is readable across segment boundaries, persisted intact, and its tracker charge returns to baseline; a short IOBuf payload persists the EOF block; an IOBuf task evicted by drop-oldest is finalized on the submitting thread and releases its tracker charge. - CachedRemoteFileReaderAsyncPeerTest (peer injected by debug point, sequential path): peer data is returned and persisted from the IOBuf payload (second read is local and peer-sourced); reordered attachments; a miss-hit-miss span bypasses peer; peer NOT_FOUND, peer error and an incomplete peer response fall back to remote storage and still submit async writes; profile timers are recorded on a peer miss and stay zero on a local read. - CrossCGAsyncWinnerRaceTest (default peer/S3 winner race): peer wins and its payload is persisted; S3 wins when peer fails; S3 wins while a slow peer is still in flight. - FileCacheProfileReporterTest covers the new profile fields. - cache_write_mode_is_resolved_for_each_read_context now expects ASYNC_WRITE with peer read enabled; async_write_keeps_direct_remote_after_peer_policy_change is removed because the behavior it covered no longer exists. Existing fetch_blocks tests pass ranges. - CachedRemoteFileReaderPeerTest.*, CachedRemoteFileReaderAsyncPeerTest.*, CrossCGWinnerRaceTest.*, CrossCGAsyncWinnerRaceTest.*, AsyncCachedRemoteFileReaderTest.*, AsyncCacheWriteManagerTest.*, InflightWriteBufferIndexTest.*, FileCacheProfileReporterTest.*, BlockFileCacheTest.cache_write_mode_is_resolved_for_each_read_context (113 tests, ASAN) all pass - Behavior changed: Yes. With both async cache write and peer read enabled, cache misses use the async write path and may be served by peers. - Does this need documentation: No

…hen misses surround local blocks Issue Number: N/A Related PR: apache#69071 Problem Summary: A peer fails the whole RPC when it misses any requested block. So on the async cache write read path, a remote span with cached or inflight blocks between its misses (miss-hit-miss) bypassed peer and was reread entirely from remote storage, while the sync path requests only its empty blocks from peer and reads the blocks in between locally. Align the async path with the sync path. When peer read applies, the tablet has a peer candidate, and the non-miss blocks inside the span are all CACHE or INFLIGHT: - Those blocks are read locally first with `_materialize_async_block` (CACHE blocks are touched in the LRU). Their bytes count as local only when all of them succeed. - Peer receives only the miss ranges. On a peer win the misses are copied into the result and their IOBufs are placed at the misses' span positions, so `_submit_async_write_tasks` is unchanged. - A storage fallback (peer failure, S3 winning the race, or the candidates disappearing after the check) still reads the whole span, from which only the misses are copied. The span keeps the existing whole-span storage read in three cases: it contains a DOWNLOADING block (the async path never waits on another downloader), it is the whole-range fallback after a side race (marked on the plan), or a local read inside the span fails; a NOT_FOUND there still triggers self-heal. When peer read does not apply, the path is unchanged. When the tablet has no usable candidate (none fetched yet, or in all-miss cooldown), reading the hits locally would only delay the whole-span storage read that follows, so the sparse read is skipped and the span is reread from storage as before this change. The check uses the new read-only `CloudWarmUpManager::has_peer_candidates()`, which, unlike `get_peer_candidates()`, does not refresh candidate expiry, reset an expired cooldown or touch bvars. Such a span also starts the background candidate fetch from FE (unless in cooldown), through the same helper `_execute_remote_read()` now uses; otherwise a tablet that only reads mixed spans would never fetch candidates. Candidates may change between the check and the read; either way the read falls back to storage, as before. Three bvars report how eligible spans use peer: - `cached_remote_file_reader_async_peer_eligible_span_count`: async-path remote spans that may use peer read (peer read enabled and not bypassed by the caller). - `cached_remote_file_reader_async_peer_no_candidate_span_count`: eligible spans with hits whose tablet has no usable candidate. This takes precedence over the span shape. - `cached_remote_file_reader_async_peer_bypass_span_count`: eligible spans with hits and candidates that read remote storage only because of one of the three shape cases above. Eligible spans made purely of misses, and sparse reads, are in neither. Spans that try peer but end on remote storage (peer miss or failure, S3 winning the race) are not counted either. Peer byte accounting in the query profile is also corrected. `collect_race_result` and `_execute_sequential_peer_read` added the whole contiguous span size to `bytes_read_from_cross_cg_peer` / `bytes_read_from_same_cg_peer`, although peer returns only the requested ranges. On the sync path, DOWNLOADED blocks between the first and last EMPTY block were therefore counted as peer bytes (an existing overcount), and the sparse async read above would count its locally read middle blocks the same way. Both now add `PeerFetchResult::bytes_read`, the byte count `fetch_blocks()` verified, read before the result is moved out; the unused `span_size` parameter of `collect_race_result` is removed. The race lambda now init-captures `empty_ranges` and `candidates`, copying each once into a non-const member that the `mutable` lambda moves into `run_peer_race`. Before, it captured the const reference parameters by value, so the members were const and the `std::move` in the lambda copied a second time. The winner race gets two test sync points, `run_peer_race::peer_succeeded` and `CachedRemoteFileReader::_execute_winner_race::s3_published`, both hit after the leg releases `race->mtx`; the peer side is moved into a small `publish_peer_success()` helper. Race tests that expect a winner now wait for the other leg's signal instead of sleeping. The S3-win copy loop of the sparse read also checks the remote buffer bound. PeerRaceS3ThreadPool runs the S3 leg of every peer/S3 winner race, on the sync and async paths alike. Its size, max_peer_race_s3_thread_num, defaults to 32, smaller than the 64 race slots of max_concurrent_peer_races, and its queue is unbounded. With more running races than threads, S3 legs queue behind each other, and raising the race slots alone makes it worse. On a two-BE cluster (both PL1 disks, SSB SF1000 lineorder full scan, async write + peer; peer-async-perf-exp1-results.md): - 64 scanners, where the race slots are not the limit (0 serial peer reads, race peak 63): 64 slots / 32 threads (E0/64/C/1..3) took 169.20 s with an S3 pool queue peak of 16; 256 slots / 256 threads (E1b/64/C/1..3) took 144.28 s (-14.7%) with a queue peak of 0. - 128 scanners: 64 slots / 32 threads (E0/128/C/1..3) took 124.24 s; 256 slots with 32 threads (E1a/128/C/1..3) took 172.51 s (+38.9%) with the queue at 103-264 and about 110 ms wait per S3 task; 256 slots / 256 threads (E1b/128/C/1..3) took 77.42 s (-55.1% vs E1a). The pool is now created with max(max_peer_race_s3_thread_num, max_concurrent_peer_races) threads, so every race slot has an S3 thread, and BE logs the effective size with both config values. Changing max_concurrent_peer_races at runtime does not resize the pool; the config comment says so. None - Test: Unit Test - New CachedRemoteFileReaderAsyncPeerTest cases: miss-cached-miss with peer winning (peer gets two ranges, middle read locally, two writes submitted, second read local); the same span with peer failing (storage supplies only the misses, no double counting); an inflight middle block; a NOT_FOUND middle block rereads the span and self-heals; a downloading middle block and the whole-range fallback bypass peer. - New CrossCGAsyncWinnerRaceTest cases: sparse peer wins the race; an eligible mixed span without candidates skips the local reads (local bytes 0, remote 12), counts no_candidate and not bypass, and starts the candidate fetch (lazy-fetch counter +1, and the fetch is waited for); the same span in cooldown starts no fetch; a DOWNLOADING middle block without candidates counts no_candidate, not bypass. - CloudWarmUpManagerPeerTest.has_peer_candidates_*: four tests (unknown or empty tablet, cooling down, cooldown expired, with candidates). Unknown tablet, empty candidates and active cooldown return false; expired cooldown and present candidates return true; after each call the candidate cache hit/miss bvars, last_access_time_ms and cooldown fields are unchanged. - The ten race tests that expect peer to win wait for peer_succeeded instead of an RPC count plus a 10 ms sleep, and race_peer_stops_when_s3_wins waits for s3_published instead of a 100 ms sleep. - Split and reordered peer responses: the mock peer service returns every requested block as uneven pieces, interleaved and out of order, for a sparse span and for an all-miss span, in attachment and protobuf-bytes modes. Each case checks the result bytes, the peer/local/remote byte counts, and the cache blocks read back directly after the async writes. With the chunk sort in take_peer_span removed for a local experiment (not committed), all four fail only on the cache read-back ("LIJK", "GHEF", "KLIJ" instead of the peer bytes) while the result bytes stay correct. - Peer byte accounting: sparse peer wins assert the CG byte counters equal the 8 bytes of the two missing blocks rather than the 12-byte span, for an async race win with a cross-CG candidate (async_race_sparse_peer_wins_around_cached_block), an async sequential read with a same-CG candidate (async_sequential_sparse_peer_counts_only_miss_bytes), and a sync EMPTY/DOWNLOADED/EMPTY race win (sync_peer_wins_counts_only_empty_block_bytes, which releases the S3 leg and waits for it to drop the reader before returning). All three read 12 before the fix. An all-miss race win still counts 12. - Before the no-candidate change the new no-candidate and has_peer_candidates tests fail to build. With only the candidate check in _prepare_async_sparse_peer removed for a local experiment (not committed), async_sparse_span_without_candidate_reads_storage_and_fetches fails on no_candidate (delta 0), bytes_read_from_local (4) and bytes_read_from_remote (8). Before the change the new sparse cases fail (no peer RPC, whole span from storage, no self-heal for the middle block) and the counter cases fail to link. - CachedRemoteFileReaderPeerTest.*, CachedRemoteFileReaderAsyncPeerTest.*, CrossCGWinnerRaceTest.*, CrossCGAsyncWinnerRaceTest.*, AsyncCachedRemoteFileReaderTest.*, BlockFileCacheTest.*, AsyncCacheWriteManagerTest.*, InflightWriteBufferIndexTest.*, FileCacheProfileReporterTest.*, CloudWarmUpManagerPeerTest.*: 318 tests, 315 pass and 3 pre-existing DISABLE_ tests are skipped (ASAN). The async peer and race cases (38 tests) pass 20 repeated runs. - Regression test (docker, cloud): cloud_p0/cache/read_from_peer/test_async_file_cache_write_peer runs two compute groups with async cache writes and the peer/S3 race. It warms CG-A until a query reads nothing from storage, lets CG-B fetch candidates, then checks on CG-B: an empty cache reads its misses from CG-A through peer (peer success, peer race wins, async submissions and eligible spans increase, no no_candidate span); after the async writes finish the same query reads neither peer nor storage; a partially warmed cache returns the same result; with peer read disabled at runtime nothing is read from peer. Every result matches CG-A. Passed on three consecutive runs (one earlier run passed every assertion but its cluster teardown exceeded the framework's 60 s compose timeout). With peer read disabled on CG-B just before the peer step (local experiment, not committed) it fails on the peer success, peer win and eligible span deltas, all 0. - Fixed an empty wait in test_async_file_cache_write: it polled async_cache_write_pending_task_count, which does not exist in be/src, so the wait never waited. It now waits for async_cache_write_buffer_memory_bytes to return to 0. The .out is unchanged and the case passed three runs. - Unit tests were run locally with GLIBC_COMPATIBILITY=OFF. - Performance: tested on a real cluster (see the Performance section of the PR description). With the default race config, fast peer source and PL1 reader: async+peer vs async+S3 −56.6% (64 scanners) / −20.0% (128), S3 GETs about −90%. - Behavior changed: Yes. With async cache write and peer read enabled, a span whose misses surround cached or inflight blocks reads those blocks locally and requests only the misses from peer. The default effective size of PeerRaceS3ThreadPool grows from 32 to 64 threads (equal to max_concurrent_peer_races), which also affects the peer/S3 race on the sync path. - Does this need documentation: No

deardeng · 2026-11-11T07:57:08Z

run buildall

### Background Async cache writes publish pending buffers to an inflight index. Reads that skip cache population can reuse those buffers when the index covers the requested range. ### What problem does this PR solve? Issue Number: None Related PR: apache#69071 (async peer cache reads) PR Type: Bugfix Problem Summary: A no-write read can crash the BE when it reuses a peer-backed inflight buffer. Use the buffer's copy interface for both heap and IOBuf payloads. ### Details - **Frequency:** Requires a read that skips cache writes and hits an inflight range containing peer data before writeback finishes. - **Root cause:** The inflight fast path calls data(), which has no valid pointer for a peer IOBuf. - **Why existing tests missed it:** The no-write coverage used heap buffers. The peer IOBuf readback coverage waited for writeback. - **Fix:** Use copy_to() for the bounded inflight copy. Gate the write worker in the new unit test so the no-write read must hit the peer IOBuf. Remove the unused fixture accessor and state left by the deleted bypass-policy test, which blocked the Clang build. - **Other paths and branches:** REMOTE_ONLY_ON_MISS uses the same copy. The regular async read path already uses copy_to(). ### Testing On devbtt, 80 ASAN tests passed across CachedRemoteFileReaderAsyncPeerTest, AsyncCachedRemoteFileReaderTest, AsyncCacheWriteManagerTest, and InflightWriteBufferIndexTest. The new no_write_reads_pending_peer_iobuf test passed. The run used GLIBC_COMPATIBILITY=OFF because the local LDB toolchain and Doris compatibility layer produced duplicate symbols at link time with the default setting. Clang-format 16, clang-tidy, and the BE build hygiene checks passed. ### Release note No-write file cache reads can reuse pending peer data without crashing the BE. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: 80 ASAN cache unit tests passed with GLIBC_COMPATIBILITY=OFF on devbtt. ### AI Disclosure - Tool: Codex - Generated: Code, unit test, and commit message - Human review: Pending

deardeng · 2026-11-12T03:48:47Z

run buildall

deardeng requested review from Gabriel39, gavinchou, liaoxin01 and morningman as code owners October 14, 2026 11:27

deardeng requested a review from bobhan1 October 14, 2026 11:27

deardeng marked this pull request as draft October 14, 2026 11:36

github-actions Bot reviewed Oct 14, 2026

View reviewed changes

bobhan1 previously approved these changes Oct 15, 2026

View reviewed changes

deardeng dismissed bobhan1’s stale review via b891898 October 20, 2026 09:28

deardeng force-pushed the peer-async-writeback branch from b891898 to 0c79ce2 Compare October 20, 2026 09:36

deardeng force-pushed the peer-async-writeback branch from 0c79ce2 to ba75f0e Compare October 20, 2026 10:45

deardeng force-pushed the peer-async-writeback branch from ba75f0e to fd80c4e Compare October 21, 2026 03:57

deardeng force-pushed the peer-async-writeback branch from fd80c4e to 8895b69 Compare October 26, 2026 03:49

deardeng marked this pull request as ready for review October 26, 2026 03:57

deardeng added the dev/4.2.x label Oct 29, 2026

deardeng force-pushed the peer-async-writeback branch from 8895b69 to aa73a10 Compare October 29, 2026 10:30

deardeng added 2 commits November 11, 2026 15:35

deardeng force-pushed the peer-async-writeback branch from aa73a10 to 2490e65 Compare November 11, 2026 07:56

deardeng requested review from Mryange and yiguolei as code owners November 11, 2026 07:56

bobhan1 reviewed Nov 11, 2026

View reviewed changes

Comment thread be/src/io/cache/cached_remote_file_reader.cpp Resolved

bobhan1 approved these changes Nov 12, 2026

View reviewed changes

deardeng added the approved Indicates a PR has been approved by one committer. label Nov 12, 2026

Conversation

deardeng commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Performance

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Oct 14, 2026

Uh oh!

deardeng commented Oct 14, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

deardeng commented Oct 26, 2026

Uh oh!

shuke987 commented Oct 26, 2026

Uh oh!

hello-stephen commented Oct 26, 2026

Uh oh!

hello-stephen commented Oct 26, 2026

Uh oh!

hello-stephen commented Oct 26, 2026

Uh oh!

hello-stephen commented Oct 26, 2026

BE Regression && UT Coverage Report

Uh oh!

deardeng commented Oct 29, 2026

Uh oh!

hello-stephen commented Oct 29, 2026

Uh oh!

hello-stephen commented Oct 29, 2026

Uh oh!

hello-stephen commented Oct 29, 2026

Uh oh!

hello-stephen commented Oct 29, 2026

BE Regression && UT Coverage Report

Uh oh!

deardeng commented Nov 11, 2026

Uh oh!

Uh oh!

deardeng commented Nov 12, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

deardeng commented Oct 14, 2026 •

edited

Loading