From 85af9757b1737aa37144201891acb1ca500a9f5b Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 6 Nov 2026 18:59:39 +0800 Subject: [PATCH] [fix](catalog) Keep the TIME scale that connectors declare ### Background Connector plugins (Hudi, and the coming Lance connector) describe their columns with the SPI's `ConnectorType`, and fe-core's `ConnectorColumnConverter` turns each into a Doris type. Connectors put a temporal column's fractional-second scale in the type's precision field: `("DATETIMEV2", 3, 0)` is `datetime(3)`, and Hudi's `HudiTypeMapping` sends `("TIMEV2", 3, 0)` for an Avro `time-millis` column and `("TIMEV2", 6, 0)` for `time-micros`. ### What problem does this PR solve? Issue Number: None Related PR: #62183 (introduced the converter without a TIMEV2 case), #64304 (moved Hudi onto it) PR Type: Bugfix Problem Summary: Since Hudi became a connector plugin, its `time-millis` and `time-micros` columns show as `time` instead of `time(3)` and `time(6)`, which is what 4.x shows and what `HudiTypeMapping` asks for: the converter drops the scale and builds a scale-0 TIMEV2. The Lance connector, ported in a follow-up, maps Arrow `time32(ms)` and `time64` the same way and would lose its scale too. The converter now reads the TIMEV2 scale from the precision field, as it already does for DATETIMEV2. ### Details - **Frequency:** Always, for every connector column whose `ConnectorType` is TIMEV2 with a scale; today that is every Hudi `time-millis` / `time-micros` column on master. Found while porting the Lance connector, by reading the converter, and confirmed by the unit test below. - **Root cause:** `ConnectorColumnConverter.convertScalarType` reads the precision field for CHAR, VARCHAR, VARBINARY, the decimals, DATETIMEV2 and TIMESTAMPTZ, but has no TIMEV2 case, so TIMEV2 falls through to `ScalarType.createType("TIMEV2")`, which returns the scale-0 `ScalarType.TIMEV2`. - **Why existing tests missed it:** Unit tests: `ConnectorColumnConverterTest` checks decimals, CHAR/VARCHAR lengths and the TIMESTAMPTZ extra info but has no TIMEV2 case, and `HudiTypeMappingTest` asserts the `("TIMEV2", 3, 0)` the plugin emits but stops before fe-core converts it. Regression: no suite under `regression-test/suites` reads a Hudi time column. - **Fix:** The TIMEV2 case uses the precision field as the scale when one is given; without one the type stays the scale-0 TIMEV2 it has always been. The fix sits in the converter because every connector goes through it. The reverse direction, `toConnectorType`, is unchanged: only the read-only Hudi and Lance connectors produce TIMEV2, and the JDBC connector maps TIME to STRING. - **Other paths and branches:** `toConnectorType` is not affected for reads (see Fix). branch-4.1 and branch-4.2 are not affected: they have no `ConnectorColumnConverter`, and their fe-core Hudi mapping (`HudiUtils`) builds `createTimeV2Type(3)` / `createTimeV2Type(6)` directly. The bug is on master since #64304. ### Testing - `ConnectorColumnConverterTest#timeScaleRidesInThePrecisionSlot` (new): fails before the fix with `expected: but was: ` and passes after; it also pins that a TIMEV2 without a precision stays scale 0. Run locally on master `f5c993f2c94` with #70338 applied, which touches neither file. - `ConnectorColumnConverterTest` (35 tests) and `HudiTypeMappingTest` (17 tests): pass locally on the same tree. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: code, tests, this description - Human review: not stated - Prompts: The author asked to port the branch-4.2 Lance FE read path to master as a connector plugin; this fix came out of that port, and the author's rule for such finds is to fix them in their own PR. Co-Authored-By: Claude Opus 5.5 --- .../converter/ConnectorColumnConverter.java | 6 ++++++ .../converter/ConnectorColumnConverterTest.java | 17 +++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java index daa288c69ecbc0..68d88357e5101c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java @@ -371,6 +371,12 @@ private static Type convertScalarType(String typeName, int precision, int scale) return ScalarType.createTimeStampTzType(precision); } return ScalarType.createTimeStampTzType(0); + case "TIMEV2": + // Like DATETIMEV2, connectors put the time scale in the precision field. + if (precision >= 0) { + return ScalarType.createTimeV2Type(precision); + } + return ScalarType.TIMEV2; case "VARBINARY": if (precision > 0) { return ScalarType.createVarbinaryType(precision); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverterTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverterTest.java index b89ff01aef6a00..0f5c3700bf6ab6 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverterTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverterTest.java @@ -230,6 +230,23 @@ void testDecimalTypeRoundtrip() { Assertions.assertEquals(6, ct.getScale()); } + @Test + void timeScaleRidesInThePrecisionSlot() { + // Connectors put the TIMEV2 scale where they put the DATETIMEV2 one: hudi maps Avro time-millis + // and time-micros to ("TIMEV2", 3|6, 0), and lance maps Arrow time32(ms) / time64(us|ns) the same + // way. Dropping it shows time(3) as time and reads the column at second precision. + Assertions.assertEquals(ScalarType.createTimeV2Type(0), + ConnectorColumnConverter.convertType(ConnectorType.of("TIMEV2", 0, 0))); + Assertions.assertEquals(ScalarType.createTimeV2Type(3), + ConnectorColumnConverter.convertType(ConnectorType.of("TIMEV2", 3, 0))); + Assertions.assertEquals(ScalarType.createTimeV2Type(6), + ConnectorColumnConverter.convertType(ConnectorType.of("TIMEV2", 6, 0))); + Assertions.assertEquals("time(3)", + ConnectorColumnConverter.convertType(ConnectorType.of("TIMEV2", 3, 0)).toSql()); + // No precision given: the scale-0 TIMEV2 it has always been. + Assertions.assertEquals(ScalarType.TIMEV2, ConnectorColumnConverter.convertType(ConnectorType.of("TIMEV2"))); + } + @Test void testWithTimeZoneColumnSetsExtraInfo() { // A ConnectorColumn marked withTimeZone() must convert to a Doris Column carrying the