[eldenmoon]

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Add a values index for VARIANT columns so that full-text MATCH can run on the whole column (the VARIANT root), not only on one subpath.

CREATE TABLE t (
    id BIGINT,
    v VARIANT,
    INDEX idx_v_values (v) USING INVERTED PROPERTIES("variant_index_scope" = "values", "parser" = "english")
) DUPLICATE KEY(id) ...
PROPERTIES ("inverted_index_storage_format" = "SNII");

SELECT * FROM t WHERE v MATCH_ANY 'timeout error';

Index contents

• One document per row; SQL NULL rows are null documents, empty objects/arrays are empty documents.
• Every scalar leaf is visited recursively through objects and arrays and its text is analyzed independently: strings as-is, booleans as true/false, integers and decimals in canonical decimal form (42, 42.0 and 42.00 all give 42), floating values in shortest form. Object keys, JSON null, date/time, binary and UUID leaves produce no text.
• The index is built from the final stored value, exactly what a scan returns for the segment: rows are reassembled with the read-side VariantAssembler in 4096-row batches after shredding, so typed-path conversion, sparse/DOC storage and hidden ancestors behave the same with and without the index.
• Terms are the analyzer's raw tokens, docs-only (no positions). The writer reuses the existing SNII array-document API.

DDL

• variant_index_scope = values on a VARIANT column, at most one per column, beside ordinary path indexes (their analyzer de-duplication is independent).
• Any tokenizing built-in parser or analyzer (custom analyzer policies are rejected); lower_case and other defaults follow ordinary analyzed indexes; support_phrase defaults to false and true is rejected.
• DUPLICATE KEY and MOW UNIQUE KEY tables; MOR UNIQUE and AGGREGATE are rejected by the existing checks.
• Requires inverted_index_storage_format = SNII. Rejected: field_pattern, NestedGroup VARIANT and declared Decimal256 paths.
• CREATE INDEX and ALTER TABLE ... ADD INDEX support existing tables for new data only, with enable_light_index_change=true and enable_add_index_for_new_data=true. Historical rowsets keep their identity and receive no index build jobs. Ordinary compaction can fill their missing index once the partition has rowsets carrying the new schema. It can be combined only with other ADD INDEX clauses that remain on the light index path.
• Full-table DROP INDEX uses the existing index deletion jobs and retains other root/path/template indexes. Partition-scoped DROP remains unsupported.
• BUILD INDEX of any index remains rejected on a table with a values index: the generic builder cannot rebuild a resident VARIANT values index when rewriting an SNII container. In cloud mode ADD requires SNII for all existing and future partitions, and a table with a values index cannot change its partition index storage format away from SNII.
• A values index is stored only in SNII index files. A segment whose index file is V2/V3, for example in a RECOVERed or RESTOREd legacy partition, is written without it and MATCH on that segment evaluates rows, as for segments written before a new-data-only ADD.

Query

• Root semantics apply only when the MATCH input is a base-table VARIANT root column that has a values index (directly or through an alias). Other VARIANT expression contexts retain their existing normalization and root-predicate restrictions. After a full DROP, root MATCH in WHERE again reports the existing unsupported-root predicate error; projection MATCH retains STRING coercion.
• Root MATCH supports MATCH / MATCH_ANY / MATCH_ALL; MATCH_ALL may match tokens from different leaves of the same row. Other operators, score() on a root MATCH (subpath score() is unchanged), a USING ANALYZER that does not match the values index, and CAST(v AS STRING) MATCH on the root are rejected. SEARCH() never uses a values index.
• While values-index metadata is present, a missing physical index, projection evaluation, or enable_inverted_index_query=false uses the same leaf-token semantics on ColumnVariantV2.

Compaction

• A values index is merged natively whenever the generic SNII eligibility check passes; otherwise its root is rewritten as a whole value and the index is rebuilt from final values. The decision is made once per compaction after changelog TTL filtering; tablets without a values index keep the existing planning order.
• Known and accepted: a native merge keeps the source tokens while compaction can promote a path's type. Indexed and index-disabled MATCH can then return different rows only when all of these hold: the same path is stored with different types in different input rowsets (a boolean and a number, an integer beyond 2^53 and a double, or FLOAT and DOUBLE); those rowsets are merged natively; and the MATCH term equals such a leaf's text before or after the promotion (for example true once it becomes 1). Other promotions keep the leaf's text, such as integer width changes and INT → DOUBLE → STRING. The promotion itself changes the stored value with or without an index ({"n":true} reads back as {"n":1}) and will be fixed separately in VARIANT, as will a materialized [] that reads differently once it moves to sparse storage.

Release note

Support a VARIANT values index (variant_index_scope = values) for full-text MATCH on the whole VARIANT column, new-data-only ADD on existing tables, and full-table DROP.

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:

  ADD/DROP validation at 1661afce6daeac755beedd317f335b48f57c0459 (ASAN runtime; the ADD/DROP change adds no BE code):

  • FE build and Checkstyle passed; 74 focused FE unit tests passed, including new-data mode, duplicate scope, mixed-change atomicity and Cloud partition-format guards.
  • Executed the entire variant_p0 directory with normal golden comparison: 201 suites, 199 initially passed, 2 failed, 0 fatal and 0 skipped. One failure was a stale legacy V1-format golden: the actual full rows matched the independent V2 golden; the runner regenerated the file and a focused normal rerun passed. The remaining OSS outfile case is unverified locally because the default placeholder credentials return 403 InvalidAccessKeyId.
  • All 19 values-index suites passed. Coverage includes historical rowset identity, new-data-only ADD without BUILD jobs, MoW updates, typed templates and sparse/DOC storage, multiple root/path/analyzer indexes, DROP and same-name recreation, plus strict indexed MATCH compared with index-disabled execution.
  • Inspected 18 compaction outputs and BE logs to confirm input/output rowset replacement, raw rebuild versus native postings merge, sparse/DOC buckets and DOC-only input becoming materialized. A full FE/BE restart also preserved index metadata, complete values and strict indexed query results.
  • Cloud runtime was not exercised; the focused FE unit tests cover its ADD guards.

• Behavior changed:

  • No.
  • Yes. New index type and root MATCH on VARIANT; ordinary subpath MATCH and non-values compaction are unchanged.

• Does this need documentation?

  • No.
  • Yes. Documentation PR to follow.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

🤖 Generated with Claude Code

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[eldenmoon]

run buildall

[eldenmoon]

/review

[github-actions]

Request changes: three P1 wrong-result cases and three P2 issues are reported inline. They cover final logical-path visibility during indexing, derived subpath MATCH coercion, visibility changes during native compaction, root/path analyzer separation during ALTER, analyzer identifier case, and accepted declared-type ingestion. No pre-existing inline findings were present.

Static review covered all 70 authoritative changed paths and their relevant producers/consumers. No builds, tests, benchmarks, source edits or submodule initialization were performed. The code-review skill was applied; its critical checkpoints conclude:

1. Goal and proof: the minimal whole-column token-search pipeline is implemented, but the three P1 cases violate MATCH semantics or query/index parity. Unit/regression tests were inspected and do not cover these triggers.
2. Scope and reuse: changes are focused on values-index creation, V2 writing/querying and compaction; existing SNII/analyzer/NULL infrastructure is reused. Logical visibility and analyzer-scope reuse need the inline fixes.
3. Concurrency and locks: no new worker threads or lock order; new mutable writer/query/planner state is operation-local. Existing cache handles and compaction publication synchronization are retained. No separate concurrency defect confirmed.
4. Lifecycle and initialization: owning writers/readers and borrowed leaf buffers survive synchronous consumption; copied schemas prevent shared-schema mutation. No cross-translation-unit static initialization dependency found.
5. Configuration: no new runtime configuration or session variable. New index properties use persisted metadata; existing analyzer case/filter settings are bound. No dynamic-setting propagation gap confirmed; the explicit English analyzer-name check needs canonical resolution as reported inline.
6. Compatibility: scope/version stamping and token bytes use existing property-map transport; unsupported layouts and old versions are rejected by the new paths. Mixed-version execution was assessed statically, not run.
7. Shared definitions: no Thrift/protobuf definition or field-ID changes.
8. Parallel paths: ordinary/DOC, root/path, scalar/index, native/rebuild, local/cloud and vertical/nonvertical flows were examined. Inline findings identify missed path-alias, ALTER and compaction-layout cases.
9. Conditions and errors: format/parser/keys/phrase/lifecycle restrictions and missing-index handling propagate explicit failures. The native safety condition and analyzer-name check are incomplete as detailed inline; declared Decimal256 schemas can also pass DDL and then fail supported inserts during index construction.
10. Test coverage: inspected BE/FE unit tests and all four changed regression suites; coverage includes NULL/empty documents, arrays, promotions, control bytes, analyzer case, native/rebuild modes and metadata sharing. Add the inline reproductions, including analyzer identifier case and values-index insertion of declared Decimal256 paths. Tests were not executed.
11. Expected results: changed expected outputs are consistent with their covered cases; quickTest sorting explains lexical ID order. Static inspection cannot establish output-generation provenance.
12. Observability: existing SNII counters and new native/rebuild decision/reason logs expose the relevant paths. No additional observability blocker confirmed.
13. Persistence and failover: index properties use existing catalog serialization, converters and CREATE journaling/replay. No new transaction protocol or visible-version rule is introduced.
14. Writes and atomicity: checked writer/merge failures abort before rowset publication and native sessions retain abort handling. The compaction visibility mismatch can pass existing row/path-count checks.
15. FE/BE propagation: analyzer metadata uses existing MATCH fields and index scope/version use existing property maps; the alias finding shows a semantic binding error rather than a missing wire field.
16. Performance and memory: metadata aggregation/eligibility reuse and row-major leaf traversal avoid repeated scans; native scratch retains cap/accounting and checked failures. No performance measurement or release-artifact ABI validation was performed.
17. Other invariants: no separate committed-transaction visibility, version, delete-bitmap or memory-lifetime defect was established. Test execution and exhaustive runtime/rolling-upgrade validation remain outside this static review.

User focus: no additional focus was supplied. All candidates and initial risks have recorded dispositions in the shared ledger; no unresolved static coverage question remains.

Review converged after 3 round(s).

[hello-stephen]

TPC-H: Total hot run time: 28622 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 61ab69fa942bc135c5cb317574a55a41e3f5e29d, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17592	5181	5098	5098
q2	2146	359	256	256
q3	10183	1412	821	821
q4	4683	474	334	334
q5	7444	801	545	545
q6	185	185	155	155
q7	757	818	606	606
q8	9253	1380	1126	1126
q9	5550	4447	4234	4234
q10	6835	1282	1073	1073
q11	436	249	222	222
q12	648	405	276	276
q13	18098	3671	3043	3043
q14	278	280	248	248
q15	q16	734	733	674	674
q17	1204	776	584	584
q18	7134	6195	6118	6118
q19	1148	961	589	589
q20	393	341	232	232
q21	5002	2129	2296	2129
q22	348	294	259	259
Total cold run time: 100051 ms
Total hot run time: 28622 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5617	5641	5641	5641
q2	701	602	534	534
q3	5231	5408	4929	4929
q4	2023	2164	1551	1551
q5	5259	5177	5814	5177
q6	247	196	146	146
q7	2327	2092	1938	1938
q8	3173	2919	2761	2761
q9	8194	7988	7992	7988
q10	3758	3668	3251	3251
q11	626	449	418	418
q12	687	704	520	520
q13	3483	4062	3333	3333
q14	300	308	295	295
q15	q16	712	705	666	666
q17	8438	7632	7082	7082
q18	12971	12154	12941	12154
q19	931	795	787	787
q20	2142	2168	1922	1922
q21	5769	4663	4788	4663
q22	494	465	387	387
Total cold run time: 73083 ms
Total hot run time: 66143 ms

[hello-stephen]

TPC-DS: Total hot run time: 151220 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 61ab69fa942bc135c5cb317574a55a41e3f5e29d, data reload: false

query5	4300	604	447	447
query6	420	211	192	192
query7	4821	445	230	230
query8	329	169	153	153
query9	8759	3223	3221	3221
query10	456	336	266	266
query11	6065	3848	3568	3568
query12	148	93	90	90
query13	1256	464	341	341
query14	6144	3916	3659	3659
query14_1	3437	3439	3473	3439
query15	211	197	183	183
query16	938	465	402	402
query17	872	670	531	531
query18	2427	428	314	314
query19	235	153	133	133
query20	87	81	82	81
query21	211	133	103	103
query22	12945	12981	12828	12828
query23	13510	12766	12537	12537
query23_1	12474	12475	12554	12475
query24	7343	848	458	458
query24_1	507	475	480	475
query25	501	398	341	341
query26	1260	253	136	136
query27	2776	439	273	273
query28	4631	2029	2046	2029
query29	1545	569	435	435
query30	295	224	188	188
query31	1028	923	809	809
query32	158	97	86	86
query33	514	294	239	239
query34	913	876	514	514
query35	737	742	642	642
query36	825	810	743	743
query37	133	101	85	85
query38	1760	1706	1630	1630
query39	793	791	749	749
query39_1	724	717	746	717
query40	216	123	99	99
query41	67	64	64	64
query42	88	84	85	84
query43	345	357	314	314
query44	1339	704	711	704
query45	177	170	163	163
query46	840	959	562	562
query47	2906	2857	2887	2857
query48	298	307	217	217
query49	573	397	327	327
query50	687	266	217	217
query51	10492	10627	10252	10252
query52	82	80	84	80
query53	186	204	148	148
query54	232	193	175	175
query55	70	67	70	67
query56	238	225	213	213
query57	1626	1615	1558	1558
query58	289	264	281	264
query59	2172	2173	2008	2008
query60	283	242	228	228
query61	145	143	141	141
query62	396	330	282	282
query63	192	156	164	156
query64	2756	950	762	762
query65	3296	3285	3265	3265
query66	1780	415	297	297
query67	20093	20104	19948	19948
query68	3017	1017	634	634
query69	385	278	313	278
query70	848	774	767	767
query71	316	238	215	215
query72	2713	2630	2321	2321
query73	515	528	305	305
query74	4652	4533	4368	4368
query75	2353	2298	1887	1887
query76	2315	1033	594	594
query77	361	383	272	272
query78	9842	9587	9160	9160
query79	901	854	536	536
query80	486	427	333	333
query81	538	314	277	277
query82	203	145	102	102
query83	199	198	177	177
query84	292	122	98	98
query85	762	420	363	363
query86	264	243	232	232
query87	2032	1949	1814	1814
query88	3714	2770	2747	2747
query89	317	282	236	236
query90	2092	177	165	165
query91	174	151	133	133
query92	102	88	88	88
query93	1006	1020	583	583
query94	453	309	298	298
query95	583	338	306	306
query96	686	487	239	239
query97	2450	2429	2335	2335
query98	158	151	150	150
query99	760	757	639	639
Total cold run time: 233602 ms
Total hot run time: 151220 ms

[hello-stephen]

ClickBench: Total hot run time: 24.33 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 61ab69fa942bc135c5cb317574a55a41e3f5e29d, data reload: false

query1	0.00	0.00	0.00
query2	0.10	0.04	0.05
query3	0.26	0.09	0.07
query4	1.61	0.08	0.11
query5	0.23	0.24	0.25
query6	1.17	0.69	0.67
query7	0.03	0.01	0.00
query8	0.06	0.04	0.04
query9	0.40	0.33	0.35
query10	0.57	0.57	0.57
query11	0.20	0.14	0.15
query12	0.19	0.15	0.16
query13	0.49	0.49	0.49
query14	1.04	1.02	1.00
query15	0.62	0.59	0.63
query16	0.35	0.35	0.36
query17	1.12	1.12	1.12
query18	0.21	0.18	0.19
query19	2.06	1.99	2.02
query20	0.02	0.02	0.01
query21	15.47	0.18	0.13
query22	4.97	0.06	0.05
query23	16.27	0.32	0.12
query24	2.65	0.55	0.49
query25	0.14	0.07	0.05
query26	1.59	0.23	0.16
query27	0.06	0.06	0.04
query28	3.83	0.80	0.37
query29	12.52	4.22	3.28
query30	0.28	0.13	0.10
query31	2.78	0.55	0.32
query32	3.26	0.60	0.49
query33	3.28	3.26	3.18
query34	15.53	4.01	3.41
query35	3.36	3.34	3.36
query36	0.57	0.43	0.44
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.15	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.05	0.03	0.04
Total cold run time: 97.8 s
Total hot run time: 24.33 s

[hello-stephen]

BE UT Coverage Report

Increment line coverage 52.94% (9/17) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	67.14% (32481/48378)
Line Coverage	52.02% (344178/661582)
Region Coverage	47.43% (279601/589472)
Branch Coverage	49.14% (132044/268716)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 52.94% (9/17) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.37% (36273/46883)
Line Coverage	62.65% (410085/654553)
Region Coverage	59.11% (347172/587344)
Branch Coverage	60.02% (160896/268064)

[eldenmoon]

run buildall

[eldenmoon]

/review

[github-actions]

Reviewed all 80 changed paths at 7eb0b6cebfe21d53ef94b8d1d1cc2f65c14bf1fc by static inspection. Request changes for the remaining instance of existing alias-lineage finding 4400052215. No duplicate or additional inline finding is posted.

The original mergeable-project alias example is repaired, but ordinary subpath values crossing UNION ALL still lose their scalar lineage. With enable_prune_nested_column=false and enable_match_without_inverted_index=true, consider:

SELECT o MATCH_ANY 'hello'
FROM (
  SELECT v['obj'] AS o FROM t
  UNION ALL
  SELECT v['obj'] AS o FROM t
) u;

The reduced analyzed plan is:

Project(MatchAny(o#5, 'hello'))
  UnionAll(o#5: VARIANT, no subpath or original column)
    Project(ElementAt(v#1, 'obj') AS o#2) -> Scan(t)
    Project(ElementAt(v#3, 'obj') AS o#4) -> Scan(t)

LogicalSetOperation.buildNewOutputs creates that metadata-free output. ExpressionAnalyzer.visitMatch omits the former STRING cast, and Match.withChildren cannot recover a subpath from the bare union slot. PushProjectThroughUnion accepts only slot/cast chains, and the alternate PushDownProject UNION rule returns unchanged when nested-column pruning is disabled. The MATCH therefore remains above the union; ExpressionTranslator.visitMatch then rejects it as a root with no unambiguous index binding. This is an ordinary subpath query that previously had scalar STRING semantics, separate from the deliberately unsupported UNION of actual root columns. Preserve scalar/subpath lineage through this boundary and cover it in regression tests. This retains the existing issue's identity rather than reposting it.

The other five existing findings are addressed at this head: finalized ordinary/sparse/DOC postings now suppress hidden ancestors; native planning rejects materialized-to-sparse transitions; ordinary ALTER ADD INDEX uses separate root/path analyzer scopes; English analyzer names are canonically resolved; and declared Decimal256 paths, including arrays, are rejected during values-index CREATE validation. Their original examples have targeted tests in the diff. Existing P1s 4400052211 and 4400052218 are not carried forward.

Critical checkpoint conclusions:

• Goal and correctness: the supported root token-search implementation and test coverage are present; the derived-subpath regression above remains.
• Scope and reuse: changes implement the stated minimal feature and reuse SNII text lowering, canonical leaf traversal, storage conversion, NULL tracking and index transport.
• Concurrency and locks: no new worker or lock order is introduced; reader-map/cache lifetimes and compaction publication retain existing synchronization.
• Lifecycle and initialization: root writers own their SNII writer, borrowed leaf bytes survive synchronous document consumption, and finish/error paths release or transfer tracked resources. No new cross-translation-unit initialization dependency was found.
• Configuration: no new server/session configuration is added; existing compaction switches select native or rebuild behavior.
• Compatibility: scope plus format version 4 gates root selection, TOKEN bytes remain 0x07 plus token bytes, and DDL rejects unsupported versions/modes. Rolling-upgrade runtime behavior was not executed.
• Shared definitions: no thrift/protobuf field or enum changes.
• Parallel paths: CREATE/ALTER/BUILD/DROP, filter/projection, root/path/SEARCH/scoring, ordinary/DOC storage, and full/cumulative/nonvertical/local/cloud compaction were traced. UNION subpath projection remains the verified exception above.
• Conditional checks: restrictions correspond to concrete unsupported query/layout/type combinations; unsafe native proofs choose rebuild rather than trusting postings.
• Test coverage: inspected FE/BE unit tests and SQL suites for visibility, NULL/empty/NOT/OR/count, analyzer case, index/scan parity, conversion, native merge and rebuild. No builds, tests or benchmarks were run in this review.
• Expected results: changed SQL outputs were inspected against their queries; this static review does not independently establish execution or output-generation provenance.
• Observability: native/rebuild decisions include index/tablet and reason; existing reader counters and the existing merge latency timer complement those logs.
• Persistence and failover: new index properties use existing persistence/serialization; no new EditLog or transaction protocol was introduced.
• Data writes and atomicity: schema extension creates a new schema; native merges preserve validated rowid coverage and existing abort-before-publication behavior. Rebuild indexes finalized storage values.
• FE/BE propagation: scope/version and analyzer settings use existing index-property and MatchPredicate channels; no new wire variable requires another send path.
• Performance and memory: source metadata is aggregated once for all roots, row-path processing avoids scanning every path for every row, and native merges have memory preflight plus hard-cap failure handling. No performance result is claimed.
• Final missed-path sweep: all candidates and risk focuses have explicit dispositions; no additional substantiated finding or unreviewed changed-file scope remains.

There was no additional user-provided focus. Review convergence is separate from the blocking verdict: the retained existing finding still requires changes. Validation is limited to static source and test inspection.

Review converged after 3 round(s).

Existing P0/P1 findings confirmed for this head: #69761 (comment)

[hello-stephen]

TPC-H: Total hot run time: 28795 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7eb0b6cebfe21d53ef94b8d1d1cc2f65c14bf1fc, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17573	5174	5068	5068
q2	2160	315	244	244
q3	10251	1410	835	835
q4	4683	461	339	339
q5	7448	816	531	531
q6	187	187	153	153
q7	744	793	608	608
q8	9268	1346	1091	1091
q9	5596	4204	4188	4188
q10	6839	1290	1039	1039
q11	427	247	229	229
q12	632	393	273	273
q13	18123	3637	3018	3018
q14	278	276	252	252
q15	q16	742	719	677	677
q17	1217	755	588	588
q18	7142	6175	6108	6108
q19	1257	954	602	602
q20	400	345	223	223
q21	6101	2655	2451	2451
q22	393	334	278	278
Total cold run time: 101461 ms
Total hot run time: 28795 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5992	5949	6181	5949
q2	759	620	611	611
q3	5670	5789	5281	5281
q4	2033	2209	1596	1596
q5	5781	5793	5645	5645
q6	240	183	138	138
q7	2300	2034	1805	1805
q8	3048	2708	2663	2663
q9	7925	7466	7450	7450
q10	3666	3603	3164	3164
q11	589	416	372	372
q12	690	663	506	506
q13	3331	3572	3066	3066
q14	288	291	266	266
q15	q16	671	676	623	623
q17	7811	7048	6955	6955
q18	12900	12130	12813	12130
q19	932	850	826	826
q20	2172	2139	1894	1894
q21	5700	4626	4833	4626
q22	509	449	423	423
Total cold run time: 73007 ms
Total hot run time: 65989 ms

[hello-stephen]

TPC-DS: Total hot run time: 151420 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7eb0b6cebfe21d53ef94b8d1d1cc2f65c14bf1fc, data reload: false

query5	4334	602	455	455
query6	423	209	197	197
query7	4800	444	241	241
query8	326	169	152	152
query9	8824	3233	3256	3233
query10	486	347	269	269
query11	5986	3877	3575	3575
query12	150	90	86	86
query13	1244	454	347	347
query14	6070	3980	3696	3696
query14_1	3444	3463	3404	3404
query15	208	201	178	178
query16	958	458	435	435
query17	879	672	535	535
query18	2422	426	317	317
query19	205	165	135	135
query20	91	83	87	83
query21	217	133	111	111
query22	13076	13009	12881	12881
query23	13503	12834	12379	12379
query23_1	12444	12636	12480	12480
query24	7376	859	490	490
query24_1	495	489	493	489
query25	549	436	337	337
query26	1262	254	141	141
query27	2764	451	277	277
query28	4612	2041	2028	2028
query29	1548	575	439	439
query30	296	210	182	182
query31	1010	928	810	810
query32	132	85	89	85
query33	522	296	249	249
query34	911	883	524	524
query35	742	752	649	649
query36	809	806	764	764
query37	125	102	78	78
query38	1780	1738	1670	1670
query39	793	789	746	746
query39_1	730	726	754	726
query40	218	117	94	94
query41	67	65	63	63
query42	87	83	80	80
query43	347	368	311	311
query44	1321	703	712	703
query45	181	184	165	165
query46	858	965	580	580
query47	3006	3079	2859	2859
query48	310	317	223	223
query49	588	408	301	301
query50	694	283	215	215
query51	10546	10182	10374	10182
query52	82	82	69	69
query53	192	209	158	158
query54	231	215	184	184
query55	70	67	65	65
query56	240	210	221	210
query57	1671	1624	1560	1560
query58	283	264	266	264
query59	2125	2148	1943	1943
query60	283	239	223	223
query61	148	144	147	144
query62	392	335	281	281
query63	194	167	162	162
query64	2740	944	767	767
query65	3330	3280	3280	3280
query66	1772	414	301	301
query67	20034	20193	20080	20080
query68	3242	1030	639	639
query69	395	292	251	251
query70	863	831	779	779
query71	307	241	222	222
query72	2821	2661	2295	2295
query73	546	539	299	299
query74	4649	4532	4358	4358
query75	2354	2291	1936	1936
query76	2401	1019	610	610
query77	371	376	291	291
query78	9609	9593	9120	9120
query79	915	876	531	531
query80	509	411	341	341
query81	511	322	267	267
query82	213	153	107	107
query83	305	200	187	187
query84	302	127	104	104
query85	835	447	362	362
query86	309	240	231	231
query87	2002	1938	1826	1826
query88	3823	2800	2778	2778
query89	322	284	243	243
query90	1909	171	175	171
query91	157	153	129	129
query92	99	88	89	88
query93	1027	1015	576	576
query94	449	311	274	274
query95	589	322	377	322
query96	693	514	249	249
query97	2455	2426	2316	2316
query98	157	152	146	146
query99	766	767	634	634
Total cold run time: 234215 ms
Total hot run time: 151420 ms

[hello-stephen]

ClickBench: Total hot run time: 24.29 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7eb0b6cebfe21d53ef94b8d1d1cc2f65c14bf1fc, data reload: false

query1	0.01	0.00	0.01
query2	0.09	0.05	0.05
query3	0.26	0.09	0.09
query4	1.60	0.10	0.10
query5	0.24	0.25	0.24
query6	1.16	0.67	0.67
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.40	0.32	0.35
query10	0.61	0.59	0.60
query11	0.21	0.15	0.14
query12	0.19	0.15	0.15
query13	0.49	0.50	0.50
query14	1.04	1.02	1.02
query15	0.64	0.61	0.61
query16	0.35	0.35	0.34
query17	1.14	1.10	1.14
query18	0.20	0.20	0.18
query19	2.10	1.99	2.00
query20	0.02	0.02	0.01
query21	15.50	0.20	0.14
query22	4.75	0.06	0.05
query23	16.26	0.33	0.13
query24	2.84	0.66	0.38
query25	0.12	0.06	0.05
query26	1.40	0.20	0.15
query27	0.04	0.04	0.03
query28	3.36	0.78	0.37
query29	12.49	4.16	3.26
query30	0.28	0.12	0.09
query31	2.77	0.56	0.33
query32	3.23	0.60	0.51
query33	3.15	3.11	3.23
query34	15.38	4.04	3.41
query35	3.37	3.33	3.36
query36	0.57	0.44	0.43
query37	0.10	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.04	0.03
query40	0.18	0.14	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.9 s
Total hot run time: 24.29 s

[hello-stephen]

BE UT Coverage Report

Increment line coverage 100.00% (17/17) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	67.14% (32484/48381)
Line Coverage	52.04% (344308/661667)
Region Coverage	47.48% (279918/589534)
Branch Coverage	49.16% (132125/268771)

[hello-stephen]

FE UT Coverage Report

Increment line coverage 74.38% (90/121) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 100.00% (17/17) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.51% (36342/46886)
Line Coverage	62.78% (411003/654638)
Region Coverage	59.21% (347783/587406)
Branch Coverage	60.14% (161240/268119)

[eldenmoon]

run cloud_p0

[eldenmoon]

run external

[eldenmoon]

run vault_p0

[eldenmoon]

run nonConcurrent

[eldenmoon]

run buildall

[eldenmoon]

/review

[github-actions]

Request changes: two previously reported P1 correctness findings still apply at head 3f8e117c1e3a8a20e88b5a967519ea4a36718217. No new nonduplicate finding was substantiated. The review converged after one round: both complete reviewers and the separate compaction-risk reviewer returned NO_NEW_VALUABLE_FINDINGS, all 94 changed paths were covered, and every candidate/risk focus has a disposition.

Existing findings retained without duplicate inline comments:

• P1: native merge retains obsolete leaf text. construct_index_compaction_columns() admits the values index through generic SNII eligibility without checking final VARIANT leaf types; plan_values_index_compaction() then leaves eligible roots on native merge. The new promotion-contract golden explicitly confirms the result: after merging {"n":true} with {"n":1}, stored rows both contain 1, but indexed MATCH 'true' still returns row 1 while row evaluation returns none; indexed MATCH '1' returns only row 2 while row evaluation returns both. The author acknowledges the limitation; the current code still exhibits the reported mismatch.
• P1: sparse-layout changes invalidate ancestor visibility. The assembler still omits a materialized empty array, while SparseColumnMergeIterator serializes non-NULL []. With the reported top-two-path scenario, moving a.b=[] into sparse storage can hide a="needle", while the native values merge retains the original needle posting. The underlying VARIANT layout transformation predates this feature, but the values-index/scanned-result divergence remains. This is already reported, so it is retained by comment ID rather than reposted.

Critical checkpoint conclusions (static inspection):

1. Goal and correctness: whole-root scalar-leaf MATCH, new-data-only ADD, and full DROP are implemented; the two existing compaction mismatches above remain blockers. Shared leaf traversal and final-row assembly address the earlier raw-leaf/hidden-ancestor writer concerns.
2. Scope, clarity and reuse: the FE validation/binding and BE storage/query/compaction changes are cohesive for this feature. The implementation reuses VariantAssembler, SNII array documents and existing index lifecycle machinery.
3. Concurrency and locks: ADD validation/publication and the cloud future-format invariant use the table mutation lock. The cloud SET/ADD race is repaired and has a deterministic latch test. Operation-owned writer/planner state and existing reader-cache locking introduce no additional substantiated race or lock-order regression.
4. Lifecycle and initialization: values writers own their SNII state, retain borrowed batch owners through assembly, and propagate failures through existing cleanup. No new thread, ownership cycle or cross-translation-unit static-initialization dependency was identified.
5. Configuration: variant_index_scope is a persistent index property. No process/session configuration is added; existing light-index/new-data flags gate ADD.
6. Compatibility: existing property serialization preserves scope/analyzers; old metadata without scope remains ordinary. V2/V3 segments deliberately omit values writers/readers and use row fallback. No new container codec or serialized column layout is introduced. Mixed-version runtime behavior was not executed; the new feature requires supporting BE code.
7. Thrift/Protobuf IDs: not applicable; no gensrc definition changes.
8. Parallel paths: CREATE/ALTER/CREATE INDEX, full/partition DROP and BUILD restrictions, cloud/local/ordered/cold/grouped compaction, horizontal/vertical/native/rebuild/retry, MoW updates, and materialized/sparse/DOC paths were traced. Ordinary path/template indexes, SEARCH and subpath scoring remain separated from root values semantics.
9. Conditional checks: the format, model, root, missing-index and rebuild branches explain concrete failure scenarios. No additional verified branch defect remained after tracing their callers.
10. Tests: inspected all changed FE/BE tests, 19 regression suites and 18 goldens. Coverage includes NULL/NOT/empty tokens, canonical numerics, analyzers, aliases, missing physical/logical indexes, legacy formats, ADD/DROP/recreate, schema changes, MoW and compaction modes. Builds and tests were not run under this static-only contract.
11. Expected results: inspected exact golden blocks and indexed/scan comparisons. Results agree with their SQL except the deliberately recorded promotion mismatch, which confirms the existing blocker. Static inspection cannot establish golden-generation provenance.
12. Observability: existing eligibility-reason logs, schema/index job history, query/compaction counters and debug points provide tracing; no distinct missing-observability defect was identified.
13. Persistence and failover: existing Index serialization and light-schema EditLog/replay carry the complete index metadata. No new transaction protocol, journal record or visible-version mechanism is introduced. Failover was not exercised.
14. Writes and atomicity: mixed values-ADD/schema clauses are rejected before live metadata mutation; pure ADD stages copied index lists. Full DROP uses existing deletion-job prechecks. Existing rowset/MoW publication and version rules remain in use; no distinct new publication defect was found.
15. FE/BE propagation: scope and analyzer information use existing Index properties and MatchPredicate fields. No new wire field or separate send path was left unhandled in the inspected flow.
16. Performance and memory: writer assembly is batched at 4096 rows; native eligibility is decided once and reuses source readers. SNII's tracked build/spill ownership is reused. No additional definite performance or memory-safety regression was substantiated; no benchmark was run.
17. Other concerns: the earlier alias, case-insensitive lookup, Decimal256, MOR analyzer, subpath-score, missing-index and legacy-format findings are repaired or unreachable as recorded in the review ledger. Removed RESTORE checks are no longer applicable.

User focus: no additional focus points were supplied. No unresolved candidate or static coverage gap remains. Runtime, cloud deployment, fault-injection and mixed-version execution were not performed; test files and goldens are inspected evidence, not fresh execution results. Review convergence does not mean the two existing P1 bugs are resolved.

Review converged after 1 round(s).

Existing P0/P1 findings confirmed for this head: #69761 (comment), #69761 (comment)

[eldenmoon]

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69761
commit: 3f8e117c1e3a8a20e88b5a967519ea4a36718217
base: 9e598d38789571a7fc30652aa054f035bd4f1184
reviewed_at: 2026-11-12T12:07+08:00
reviewer: eldenmoon
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

• Scope follows the request: exclude only discussion_r4400052218 (materialized-empty-array to sparse ancestor-visibility drift) and discussion_r4478543524 (canonical leaf-text drift under native type promotion). These remain declared known limitations; this receipt does not claim they are fixed. Finding counts apply to the remaining reviewed scope.
• Reviewed all 94 changed paths and 20 commits with two complete, explicitly qualified coverage reviewers plus lead verification. No additional actionable finding; coverage 94/94, no open candidate, converged in round 1. R1/R2/R3 were fully resolved by source evidence; redundant risk dispatches were dropped under skill section 5.1.
• Static review only: no local build, unit/regression suite, benchmark, live cloud or rolling-upgrade execution. Exact-head CI observed compile/cloudut success; feut/beut and p0/cloud_p0/vault_p0/external/nonconcurrent pending. The existing automated code-review failure retains only the two excluded items; this is not an all-CI-green assertion.

_{Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.}

[eldenmoon]

run p0