Skip to content

Commit 62fabfd

Browse files
UnamedRusclaude
andcommitted
Update contrib/datasketches-cpp to 5.2.0 and backport the HLL union fix
Backport of ClickHouse/ClickHouse@a067567, re-created here rather than cherry-picked so that it carries a sign-off. The submodule tracked `apache/datasketches-cpp` directly, pinned to `76edd74f` (2024-05-16), an upstream development commit. It moves onto a `ClickHouse/`-prefixed branch of our fork, the way `docs/development/contrib` asks: ClickHouse/datasketches-cpp, branch ClickHouse/5.2.0 de8553ba 5.2.0, the newest upstream release (2025-01-15) 23bd9b07 backport of apache/datasketches-cpp#512 apache/datasketches-cpp#512 is the HyperLogLog union fix. No upstream release carries it, so it is cherry-picked onto the `5.2.0` tag there. `uniqApacheHLL`, added next, needs it: without it a merged HLL sketch reports a wrong estimate and serializes a state that other DataSketches implementations read differently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
1 parent 66cb75e commit 62fabfd

4 files changed

Lines changed: 25 additions & 2 deletions

File tree

‎.gitmodules‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,7 @@
154154
url = https://github.com/ClickHouse/NuRaft
155155
[submodule "contrib/datasketches-cpp"]
156156
path = contrib/datasketches-cpp
157-
url = https://github.com/apache/datasketches-cpp
157+
url = https://github.com/ClickHouse/datasketches-cpp
158158
[submodule "contrib/yaml-cpp"]
159159
path = contrib/yaml-cpp
160160
url = https://github.com/ClickHouse/yaml-cpp
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
16 16 30
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
-- Tags: no-fasttest
2+
-- - no-fasttest -- compiled w/o datasketches
3+
4+
-- Apache DataSketches can serialize a Theta sketch in a compressed form (serialization
5+
-- version 4) that packs the retained values at a variable number of bits per entry.
6+
-- ClickHouse always writes the uncompressed form, but it has to read the compressed one,
7+
-- because an `AggregateFunction(uniqTheta, ...)` state can come from another
8+
-- implementation of the library. Two of the unpacking routines decoded it incorrectly,
9+
-- and the result was a silently too low estimate rather than an error.
10+
11+
-- Both states below are canonical compressed encodings of a sketch that retains 16 values
12+
-- with theta = 1, so the estimate is exact. The first packs its values at 33 bits per
13+
-- entry and the second at 35; the two widths are decoded by separate routines. The two
14+
-- sketches have exactly two values in common, so their union retains 30.
15+
16+
WITH
17+
CAST(unhex('4B01040321011ACC9310000001F4000000FA0000007D0000003E8000001F4000000FA0000007D180000000000001F4000000FA0000007D0000003E8000001F4000000FA0000007D0000003E8') AS AggregateFunction(uniqTheta, UInt64)) AS state_33_bits,
18+
CAST(unhex('4F01040323011ACC93100000007D0000000FA1000000004000000000000007D0000000FA0000001F40000003E80000007D0000000FA1000000000000003E80000007D0000000FA0000001F40000003E8') AS AggregateFunction(uniqTheta, UInt64)) AS state_35_bits
19+
SELECT
20+
finalizeAggregation(state_33_bits),
21+
finalizeAggregation(state_35_bits),
22+
finalizeAggregation(uniqThetaUnion(state_33_bits, state_35_bits));

0 commit comments

Comments
 (0)