dolt 2.5.0

[BrewTestBot]

Created by brew bump

---

Created with brew bump-formula-pr.

Details

release notes

# Merged PRs
dolt

12130: docker: bypass provisioning on cluster standbys and fail fast on invalid init SQL

Fix dolthub/dolt#12115
12128: docker: skip entrypoint setup on container restart when initialized

Fix dolthub/dolt#12115
12108: kvexec: allow NULL in the constant lookup key of an index lookup join

Fix dolthub/dolt#12106
12092: Authenticate cluster role headers before applying transitions

Fixes #12090.
12091: Fix standalone remotesrv path traversal

Reject traversal and malformed repository paths in RPC requests and sealed HTTP file requests.
Use a rooted filesystem for standalone remote store directory creation, and verify the resolved store path stays inside the configured root.
Validate upload table file IDs before converting them to hashes.

Fixes #12089.


12088: go: gcctx: GCSafepointController: Explicitly support concurrent Waiters.

This adds support for concurrent GC runs against different databases on the same server. Each GC run captures the outstanding sessions that are present when it starts and blocks on visiting only their roots. The sessions themselves can keep track of more than one outstanding waiters.

Explicit semantics are added for sessions which are themselves within a GC Safepoint Waiter, in order to prevent a potential deadlock. While the Waiter is outstanding, new Waiters which come to wait on that session can immediately visit its roots, just like the sessionAwareSafepointController would have done for that session itself when creating its own Waiter.
12086: merge_base overhaul

Add new dolt_merge_bases() table function, allowing for multiple merge bases
Add --all flag to dolt merge-base cli operation.
Existing dolt_merge_base() behavior remains unchanged.

A significant amount of testing git and it's merge-base --all behavior led to the testing harness added here. the go/libraries/doltcore/commitgraph/merge_bases_test.go is effectively the git behavior of

merge base codified.

Behavior of merge-base with shallow clones (ghost commits) was incomplete, and errored in situations it should not have. This area is much more thoroughly tested now.

Related: Merge and three-dot diff use only the tallest of several merge bases, which breaks the schema-branch pattern dolthub/dolt#12050


12077: schema, diff, dtables: keep history across same-key table recreate

Diff accepts retagged primary keys (same names and types, new column tags) instead of warning, so history survives a recreate.

Fix dolthub/dolt#12067


12075: Stop embedded SQL engine workers and clean up primary binlog streams

Closing an embedded SQL engine now stops and joins its branch-activity tracker and primary binlog manager, including when engine construction fails. Primary shutdown interrupts blocked replica writes and waits for active streams; disconnecting a replica preserves other streams, and stream exit closes the current rotated binlog file.

Based on #12068 by Alex Giurgiu; the original contributor commits and authorship are preserved.
12066: Stop keyless index producers on Close

Closing a keyless secondary-index scan before it is drained could leave its producer goroutine blocked on a full channel. The scan state and its expanded duplicate rows then stay reachable after the caller stops reading.

Close now cancels an iterator-owned child context and waits for the producer, without cancelling the caller's context. No public API, SQL syntax or configuration changes.
Test
TestKeylessIndexIterCloseStopsProducer inserts ten identical keyless rows, reads one, and closes the iterator inside testing/synctest, which fails if the producer is still blocked when the test ends. It fails without the fix and passes with it.
cd go && go test -run '^TestKeylessIndexIterCloseStopsProducer$' -count=1 ./libraries/doltcore/sqle/index
The test shows the producer stops. That Close also waits for it comes from the code (eg.Wait()), not from the test.
Example
In a manual check (not part of this PR) with a keyless table k(a BIGINT, b BIGINT), index idx_b(b) and 20 rows per value of b:

SELECT a FROM k FORCE INDEX (idx_b) WHERE b = 3 LIMIT 1 left one blocked producer before the fix and none after.
A correlated subquery with LIMIT 1 over 50 outer rows left 50 blocked producers before the fix and none after.

Checked

The index package tests, normal and race; the new test ran 50 times under race.
TestIndexes, TestIndexPrefix and TestGeneratedColumns, normal and race.
go vet, copyright headers and check_fmt.sh.

In one of our environments, TestQueries/select_length(compress(repeat('a',_1000))) fails (expected 24, actual 21) on both main and this branch. It is outside the index path. We did not run the full Go or SQL engine suites, and we have not measured how often this happens in production.


Generated by Perfloop. Human sponsor: Tomás Senart.
12062: gitauth: fix Windows FETCH_HEAD lock contention, dedup write fetches, and omit auth hints for file errors

Fix dolthub/dolt#11904
12061: dbfactory: init the git remote cache repository with the remote's object format

Fix dolthub/dolt#12060
12057: SyncRoots: return the source root read error instead of reporting success

Fixes #12056.

Close #12072

SyncRoots returned nil when srcDb.NomsRoot failed, so DOLT_BACKUP sync, sync-url and restore reported success without transferring the source root. This returns the wrapped error instead, matching how the destination root read just below already handles its error.
12052: merge: detect conflicting generated column additions

When both branches add the same generated column, report a schema conflict if its AS (...) expression or VIRTUAL/STORED mode differs, instead of silently keeping one definition. Identical additions still merge cleanly.

Compare these properties in both column equality methods and add merge tests for each case.

Fixes #11367
12049: more dolt optimizations to help dumbodb

Three smallish changes:

capture hash of object rather than recalculating
allocate in one shot in ReadBytes
reuse unchanged leaves in adaptive encoding


12047: go/store/nbs: journal_file.go: Use .bss allocated zeros slice for the write ahead.

This lets us write the zeros without interacting with allocation or GC and without the allocated array needing to RSS (outside the VMM mapping overhead).
12041: Return ErrTableAlreadyExists for existing nonlocal tables

Fix dolthub/dolt#12039
12037: bats: qualify pk in auto_increment_lock_modes check query

test1 and ranges both have a pk column, so the unqualified min(pk) and max(pk) are ambiguous. go-mysql-server b4f13d62c36d rejects them, failing the test on every run since #12035.
12034: Accelerate lookup joins with SQL row inputs

Accelerates lookup joins with SQL row inputs using shared iteration and type-aware key conversion.

Fixes Query is 1.76x slower in Dolt than MySQL dolthub/dolt#11636

Companions: Preserve numeric overflow handling in lookup joins dolthub/doltgresql#3554 and Fix extended-type hash key conversion dolthub/go-mysql-server#3997.
12031: perf: Use fdatasync on linux with a padded journal

fdatasync avoid unnecessary metadata updates to the file system, resulting in ~25% improvement in write performance.
12027: dolt version: cache failed release checks to prevent hanging

Add 3s HTTP timeout for latest release fetch.
Cache failed checks under version check file to avoid a second HTTP timeout.

Fix dolthub/dolt#12021


11928: Support atomic multi-branch transactions and commits

Enables atomic updates across branch working sets and heads with dolt_commit_all and the dolt_multi_branch_commit setting.

Companion Doltgres PR: Update Dolt and skip incompatible transaction tests dolthub/doltgresql#3615

go-mysql-server

4041: Clear sign on negative zero during DecimalMod

Fixes dolthub/dolt#12124

This is done in DecimalDiv, but it seems like the step was skipped for DecimalMod.
4040: Fix off-by-one error when finding aggregated column during replaceSortedMinMax

Fixes #4037
4034: Set orderByScope.cols properties for SetOps using that of the merged columns

Fixes dolthub/dolt#11714

Based on #3977
4032: Fix nullsafe logic for tuples containing null values

Previously, left and right elements in tuples were converted to a common type before comparing.

For DATETIME types, certain values would be converted to NULL when they did not need to be converted at all.

Fixes: Fix null safe equals for tuple comparisons dolthub/dolt#12109
4029: sql: Server lifecycle, background work and test harnesses support EngineVariables.

Part of a body of work to remove the global SystemVariables and StatusVariables registries and make them state on the Engine itself.

Server lifecycle, background work and test harnesses support EngineVariables.
Migrate remaining engine variable consumers.
Keep global-only variables live outside session overlays.
Bind collation wire harness sessions to their engines.
Make COMPRESS assertions independent of the zlib byte stream.


4028: sql: SQL planning and execution access EngineVariables through Context/Session.

Part of a body of work to remove the global SystemVariables and StatusVariables registries and make them state on the Engine itself.
4027: sql: Engine can carry EngineVariables and they are exposed on the Context and Session.

Part of a body of work to remove the global SystemVariables and StatusVariables registries and make them state on the Engine itself.
4026: sql: Add factories for independent registries of SystemVariables and StatusVariables.

Part of a body of work to remove these globals and make them state on the Engine itself.
4024: Fix panic for JSON_PATHs pointing to null objects and null slices

This PR adds regression tests for JSON_EXTRACT where the JSON_PATH points to a nil object and attempts to slice into it.

Depends on: Fix panic for NULL object and NULL slice dolthub/jsonpath#8

Fixes: JSON_EXTRACT('{"a": null}', '$.a[0]') panics dolthub/dolt#12103
4022: sql/types: build the JSON_EXTRACT result array only from paths that match

Fix dolthub/dolt#12107
4021: Do not remove GroupBy node when replacing sorted MIN/MAX aggregates

Based on #4014

Fixes dolthub/dolt#12093

MIN/MAX aggregates without an explicit GROUP BY should still return a single NULL row when no rows match. For such cases when MIN/MAX was aggregating over a primary index, we were returning no rows; this was because in replaceAgg, we were removing the GROUP BY node, which was still needed to return the correct results via the groupByIter.

This PR does the following:

renames replaceAgg to replaceSortedMinMax to more accurately reflect what the rule does
preserves GROUP BY node
have replaceSortedMinMax return a LIMIT 1 over an IndexedTableAccess instead of a LIMIT 1 ORDER BY node. Because the GROUP BY node is preserved, the replaceIdxSort rule no longer replaces the previously created LIMIT 1 ORDER BY node with a LIMIT 1 over an IndexedTableAccess. Instead, replaceIdxSortHelper must be called directly within replaceSortedMinMax.


4020: Handle locate case when str length is 0 and position is greater than 0

fixes: LOCATE panics on empty string dolthub/dolt#12102
4015: Support conditional column drops in ALTER TABLE plans

Carries column-level IF EXISTS into drop plans and skips missing columns during schema validation. Adds coverage for existing/missing columns, repeated drops, and missing-table errors.

Uses Vitess main following the merge of dolthub/vitess#501. ALTER TABLE and analyzer tests pass. Part of dolthub/doltgresql#3536.
4014: Return a NULL row for MAX/MIN without GROUP BY when no row matches

MAX/MIN without GROUP BY returned no row instead of one NULL row when no row matched, whenever replaceAgg could read the aggregated primary key column in index order:
SELECT MAX(i) FROM emptytable;                            -- no row
SELECT MAX(pk3) FROM three_pk WHERE pk1 = 2 AND pk2 = 0;  -- no row since #3907
replaceAgg replaced the GroupBy with Limit(1) over a Sort, so empty input produced no row. It now keeps the GroupBy and puts the Limit(1) over the Sort beneath it, so the scan still reads a single row when the index provides the order. replaceIdxSort doesn't descend into a GroupBy, so replaceAgg converts that Sort to an index scan itself. The 13 changed plans are the existing MAX/MIN plans, each gaining the GroupBy.

The empty-input queries added to QueryTests fail without the fix. Reported in Ungrouped MAX/MIN over a primary-key column returns no row when nothing matches dolthub/dolt#12093.
4012: Match MySQL Punctuation for DATETIME string parsing

This PR drops the regexes used for DATETIME string parsing in favor of regular logic.

The new implementation covers many missing warnings, addresses several edge cases, and handles additional delimiters missing from the previous implementation.

Additionally, fixes some wire tests that were previously silently failing.

Fixes:

Throw warning for non-standard DATE and DATETIME delimiters dolthub/dolt#11750
extend delimited datetime regex(es) dolthub/dolt#11939


4009: End non-recording spans in NewSpanIter.
4008: Make Handler.doQuery start and end the root query span. Remove Context.RootSpan and deprecate WithRootSpan.
4007: Stop ending the query in the process list when the top-level row iter is closed. The caller of BeginQuery owns EndQuery.
4005: Generate correct GetField expressions when optimizing joins on table aliases with pruned columns.

This PR fixes two related but distinct panics.

We had some duplicated logic when optimizing joins, where each implementation had a different bug that was fixed in the other.

In convertSemiToInnerJoin, we generated GetField expressions that used the table name of the underlying table, not the table alias. This could cause these expressions to not resolve correctly.

In convertAntiToLeftJoin, we used the pruned table schema instead of the full table schema when calculating ColumnIds, resulting in a potential out-of-bounds access.

Dolt PR: [no-release-notes] Bump GMS dolthub/dolt#12082

Doltgres PR: [no-release-notes] Bump Dolt dolthub/doltgresql#3589
4003: Estimate a join of two empty estimates as empty on every architecture

A join whose sides both have a row count and a distinct count of 0, as filtered reads of tables without statistics are estimated, made the selectivity 1/0 = +Inf and the cardinality 0 * +Inf = NaN. Converting NaN to uint64 is implementation-defined: arm64 gives 0, amd64 gives 1<<63. So on amd64 such a join was estimated at 2^63 rows, a hash or lookup join over it was priced in the 1e16s, and a nested-loop join won; on arm64 the same statement planned a hash join.

The distinct count is now at least 1, so the cardinality is 0 on both, as arm64 already estimated it. The test fails on amd64 without the fix.
4001: Preserve aggregate input column IDs for HAVING

Preserve GetField column IDs when adding aggregate inputs so HAVING retains grouping columns and evaluates the correct values. Fixes the incorrect result exposed by #3797.

Adds the minimal repro and related ordinary/prepared regressions, with a TODO to migrate them after #3990. Engine regressions and planner, analyzer, row execution, and aggregation tests pass; SQL expectations match MySQL and PostgreSQL.
4000: Have nested stored procedures reuse existing transaction

Nested stored procedure calls should not create a new transaction. Instead their changes are applied to the existing transaction.

Fixes: Stored procedure that CALLs another procedure inside an explicit transaction silently discards the caller's earlier writes dolthub/dolt#11950
3998: Prevent subquery filter pushdown from substituting set-returning functions into scalar predicates

Filters on SRF outputs now remain above row expansion, while safe predicates still push down and preserve correlation tracking.
3997: Fix extended-type hash key conversion

Fixes extended-type hash joins by converting keys into the comparison type.

Companions: Accelerate lookup joins with SQL row inputs dolthub/dolt#12034 and Preserve numeric overflow handling in lookup joins dolthub/doltgresql#3554.
3995: Fix SIGN to handle signed floating points < 0.5 and string truncation

Fix dolthub/go-mysql-server#3988
3994: Improve TIME parsing

This PR further improves TIME parsing/conversions to more closely match MySQL's behavior.

fixes: truncate strings when casting to time dolthub/dolt#10000
3993: Moved the external function provider into the builder overrides

This moves the ExternalFunctionProvider so that integrators can set it via overrides. We also add a call to it so that integrators are able to use functions from their provider in column defaults.
3992: Added view recursion detection

This fixes an OOM issue that occurs from recursive views.
3991: Optimize wildcard-only LIKE filters

Wildcard-only LIKE filters such as LIKE '%' and LIKE '%%%' now simplify to IS NOT NULL for string operands. The rewrite only runs when LIKE is the full filter expression, preserving NULL behavior in nested expressions.

This also covers non-UTF-8 strings and constant-folded patterns.

Tests:

go test -tags=gms_pure_go ./sql/expression -count=1
go test -tags=gms_pure_go ./sql/analyzer/... -count=1
go test -tags=gms_pure_go ./enginetest -run '^TestCharsetCollationEngine$' -count=1

Fixes #3943


3989: Infer precision for TIME function

based off of: infer fractional seconds precision for TIME function dolthub/go-mysql-server#3986

partially addresses: TIME() function infers precision based off input dolthub/dolt#11944
3985: Add regression test for ValueRowIter

This PR adds a flag to QueryTests that asserts that the generated row iterator supports the ValueRowIter interface. This interface is typically more performant than the traditional RowIter, so we want to make sure that we don't regress on queries that use it.

Dolt PR: [no-release-notes] Bump GMS dolthub/dolt#12029

Doltgres PR: [no-release-notes] Bump Dolt dolthub/doltgresql#3549
3977: Sort set operation ORDER BY by the output column

Fixes dolthub/dolt#11714

A set-op ORDER BY that named a projected column re-evaluated the left branch's expression, rather than reading the set op's output column. buildSortConditions only rewrites aliases, so an unaliased literal, or a function such as abs(-1000), sorted by a constant. This change points those terms at the output column, typed with the merged column type. Terms that are already a GetField are left unchanged, so existing plans don't change.

The expected results in the new script test come from MySQL 8.4.11. Without the fix, 10 of the 16 assertions fail. The rest are guards, including select 10 union select '9' order by 1, which has to keep sorting as strings. go test ./sql/... ./enginetest/... passes.

There are two related problems on main that I left out of scope:

select a from t union select 'x' order by 1 still sorts the table column with its own int type. MySQL compares as strings here.
select a, a from t union select b, 1 from u order by 2 sorts by column 1, because both projections share one column id.


3916: Prune table columns beneath HAVING

Allow projection pruning to traverse HAVING while retaining the columns needed by grouping, aggregates, and joins. HAVING expressions with subqueries continue to keep the outer table unpruned.

Part of dolthub/dolt#11923
3893: Fix window expression identity collisions

Use a structural identity when deduplicating window expressions so distinct resolved columns with colliding rendered names are retained independently.

The identity includes expression structure and resolved field IDs plus window partition, order, NULL-ordering, and frame semantics, while excluding the assigned root output ID.

Fixes dolthub/dolt#11855
3891: Fix implicit peer frames for window functions

Implicit ordered LAST_VALUE, BIT_AND, BIT_OR, and BIT_XOR did not consistently use the default RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW peer semantics.

Use the resolved peer group for LAST_VALUE and the shared aggregate default-frame path for the bitwise aggregates. Regression coverage includes peer rows, the exact running BIT_OR case, explicit-equivalent frames, descending order, and compound ordering. The new assertions match MySQL 8.0.45 and PostgreSQL 15.17.

Fixes dolthub/dolt#11856

Fixes dolthub/dolt#11857
3890: Fix nullable RANGE frame boundaries

Nullable numeric RANGE frames could include unrelated rows when NULL boundary comparisons disagreed with the window's ordering. Preserve the effective NULL placement in RANGE framers and use it when scanning boundaries, supporting both MySQL's NULL-first and PostgreSQL's NULL-last ascending defaults.

Regression coverage checks bounded nullable frames and explicitly verifies sorted rows and peer intervals for all four direction/NULL-rank combinations. The shared SQL results match real MySQL 8.4.11 and PostgreSQL 15.17.

Fixes dolthub/dolt#11858
3797: Resolve correlated aggregates at their outer scope

Correlated aggregates in nested queries could be assigned to the syntactically inner query even when their arguments belonged to an outer query. This could evaluate an aggregate in the wrong scope and produce runtime type errors.

This change determines aggregate ownership from resolved argument columns instead of AST nesting. It:

Tracks semantic query blocks separately from transient builder scopes.
Resolves aggregate arguments through the query-block chain while preserving normal name resolution and shadowing.
Registers each aggregate with the closest query that supplies one of its arguments.
Replaces correlated argument dependencies with the resulting aggregate dependency.
Applies the same ownership rules to GROUP_CONCAT and its ordering expressions.
Centralizes aggregate registration and dependency collection to keep IDs and references consistent.

Coverage includes ordinary and prepared execution, nested and parenthesized queries, joins, CTEs, unions, alias shadowing, mixed local and outer arguments, ordered GROUP_CONCAT, and strict-grouping failures.

The related grouped-window validation gap remains tracked in #3856.

Fixes #3814



vitess

501: Represent DROP COLUMN IF EXISTS in the DDL AST

Parses and formats ALTER TABLE ... DROP COLUMN IF EXISTS, with a column-level flag separate from table-level IF EXISTS. Adds parse/format round-trip coverage; the full parser test package passes.

Part of dolthub/doltgresql#3536.
500: Preserve subquery input expressions in EXPLAIN queries

Previously, because Explain parse nodes did not implement WalkableSQLNode, Vitess would not walk them to populate input expressions on subqueries. This let to cases where go-mysql-server would produce a different plan for a DESCRIBE PLAN query compared to the original query.

Closed Issues

12115: dolt-sql-server image: a demoted cluster node can't restart (entrypoint setup is rejected on a standby, and the exit is silent)
12124: negative dividend in mod yields incorrect negative results
11714: Top-level ORDER BY does not seem to be applied to unions
12109: Fix null safe equals for tuple comparisons
12106: Lookup join panics on a NULL literal key for a NOT NULL column
12107: JSON_EXTRACT panics when one of several paths matches nothing
12093: Ungrouped MAX/MIN over a primary-key column returns no row when nothing matches
12103: JSON_EXTRACT('{"a": null}', '$.a[0]') panics
12102: LOCATE panics on empty string
11505: Date-valued window arithmetic appends a zero-time suffix
7690: Support MySQL datetime parsing behavior
10399: TIMESTAMPDIFF should return null for invalid datetimes
12067: dolt diff cannot track beyond a 'drop & recreate' action with identical index definition
11750: Throw warning for non-standard DATE and DATETIME delimiters
11939: extend delimited datetime regex(es)
12060: git+https remote push fails with 'does not support this repository's hash algorithm' against SHA-256 git repos
12089: Standalone remotesrv Table-File Upload Endpoint Allows Unauthenticated Path Traversal File Writes on Dolt
12090: Dolt Cluster Replication Endpoint Processes Client-Controlled Role Headers Before JWT Authentication, Allowing Persistent Unauthenticated Primary Demotion
12056: SyncRoots returns success when reading the source root fails (DOLT_BACKUP sync/restore report success)
11367: Merge silently picks one expression when branches add the same generated column with different AS(...) definitions
11904: git+https remote: Windows push fails on FETCH_HEAD 'Permission denied'; redundant fetch fan-out, no --no-write-fetch-head, console per git spawn
11636: Query is 1.76x slower in Dolt than MySQL
11950: Stored procedure that CALLs another procedure inside an explicit transaction silently discards the caller's earlier writes
12039: CREATE TABLE IF NOT EXISTS fails for an existing nonlocal table on a non-target branch
10000: truncate strings when casting to time
12036: DESCRIBE PLAN {{query}} can result in a different plan than {{query}}
11858: Nullable numeric RANGE bounds expand to the whole partition
11855: Dolt merges two different window operands whose qualified-column strings collide
11857: Dolt BIT_OR ignores ORDER BY when selecting its window frame
11856: Dolt LAST_VALUE uses a ROWS-like implicit frame instead of the default RANGE peer frame
12021: dolt version blocks for 30 s or more on every call when api.github.com is slow or unreachable
4037: Panic index out of range in replaceAgg for ungrouped MIN/MAX of a non-key column when WHERE fixes the whole primary key
3943: Simplify col LIKE '%' to col IS NOT NULL in analyzer
3988: SIGN() returns 0 for non-zero values with magnitude below 0.5
3814: Plan correlated aggregates correctly across nested semantic scopes

View the full release notes at https://github.com/dolthub/dolt/releases/tag/v2.5.0.

---

Footnotes

1. replaceIdxSort does not work for GROUP BY nodes due to how it can be nested inside and/or around an ORDER BY, i.e. Sort, node. ↩

[github-actions]

🤖 An automated task has requested bottles to be published to this PR.

Caution

Please do not push to this PR branch before the bottle commits have been pushed, as this results in a state that is difficult to recover from. If you need to resolve a merge conflict, please use a merge commit. Do not force-push to this PR branch.