Repository navigation
[improvement](stream) List table streams as BASE TABLE, not VIEW - #70560
Open
morningman wants to merge 1 commit into
Open
morningman wants to merge 1 commit into
morningman wants to merge 1 commit into
Conversation
### Background Since apache#61382, a table stream (`CREATE STREAM s ON TABLE t`) is listed with the tables and views of its database in `SHOW FULL TABLES` and `information_schema.tables`, with `ENGINE` `Stream`. Incremental refresh (apache#62606) creates one for each base table of an incrementally refreshed materialized view, named `__doris_ivm_stream_<mv id>_<hash>`. `TableIf.TableType.toMysqlType` gives a stream the table type `VIEW`. ### What problem does this PR solve? Issue Number: None Related PR: apache#61382, apache#62606 PR Type: Enhancement Problem Summary: Metadata clients take a `VIEW` row for a view: they read its definition and parse it for lineage. A stream has no view definition; `SHOW CREATE TABLE` prints `CREATE STREAM ... ON TABLE ...`. OpenMetadata, for one, ingests every stream as a view whose definition is that statement, and the streams incremental refresh creates appear as views nobody created. This PR lists a stream as `BASE TABLE`: it is read with SELECT like a table, and MySQL defines no other table type that fits. `ENGINE` stays `Stream`, so clients can still tell streams apart. Table streams and incremental refresh are only on master, so no released version changes its output. ### Details - **Before and after:** | | Before | After | |---|---|---| | `Table_type` of a stream in `SHOW FULL TABLES` | `VIEW` | `BASE TABLE` | | `TABLE_TYPE`, `ENGINE` of a stream in `information_schema.tables` | `VIEW`, `Stream` | `BASE TABLE`, `Stream` | | the `__doris_ivm_stream_*` streams of an incrementally refreshed MV | listed, `VIEW` | listed, `BASE TABLE` | Views keep `VIEW` and system tables `SYSTEM VIEW`. `information_schema.views`, `SHOW STREAMS`, `SHOW CREATE TABLE` and `information_schema.table_streams` are unchanged: they select streams and views by `TableType`, not by this string. - **Configs and defaults:** None. - **Impact on existing users:** None on released versions: table streams and incremental refresh are on neither branch-4.1 nor branch-4.2. On master, `SHOW FULL TABLES ... WHERE Table_type = 'VIEW'` and JDBC `getTables` with the type `VIEW` no longer return streams, and the type `TABLE` now does. Nothing is persisted: the type is computed whenever a listing is served. ### Testing - Regression test (new): `query_p0/schema_table/test_stream_table_type` creates a table, a stream on it and a view, and checks the stream and the view in `SHOW FULL TABLES`, all three in `information_schema.tables`, and that `information_schema.views` lists only the view. The `.out` is generated. On the code before this PR it fails at its first check: `Expect cell is: BASE TABLE`, `But real is: VIEW`. - FE unit test (new): `TableTypeTest` checks that a stream maps to `BASE TABLE` with the engine `Stream`, and that `VIEW`, `INLINE_VIEW`, `SCHEMA`, `OLAP` and `MATERIALIZED_VIEW` keep their types. `./run-fe-ut.sh --run org.apache.doris.catalog.TableTypeTest`: 2 tests, 0 failures. - Manual: an incrementally refreshed materialized view over a merge-on-write table with changelog; its `__doris_ivm_stream_*` stream is listed as `BASE TABLE` / `Stream` in both listings, and as `VIEW` / `Stream` before this PR. - Environment: a local cluster whose FE ran this branch's fe-core jar (and, for the before runs, a jar of the same base without this change), with a BE from an earlier master build (`6991e766329`); the change is FE-only. ### Notes for Reviewers - Why `BASE TABLE` and not a new value such as `STREAM`: MySQL defines `BASE TABLE`, `VIEW` and `SYSTEM VIEW`, and clients map and filter by them (MySQL Connector/J returns `BASE TABLE` rows as `TABLE` from `getTables`), so a new value would drop streams from client listings or break their type mapping. - Why the streams incremental refresh creates stay listed: they are real objects that `SHOW STREAMS` and `information_schema.table_streams` show and that can be queried; hiding them by their name prefix in some listings would make the listings disagree. - The incremental-refresh case is checked by hand rather than in the suite: it needs a merge-on-write base table with changelog, and a materialized view's `ENGINE` in `information_schema.tables` is about to change (apache#70523), which the `.out` would then depend on. ### Release note None ### Check List (For Author) - Behavior changed: Yes. Table streams are listed as `BASE TABLE` instead of `VIEW`; table streams are not released yet. - Does this need documentation: No. doris-website has no page on table streams yet. - Pre-submission status: Ready ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: the code change, both tests and this description - Human review: not stated Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Since #61382, a table stream (
CREATE STREAM s ON TABLE t) is listed with the tables and views of its database inSHOW FULL TABLESandinformation_schema.tables, withENGINEStream. Incremental refresh (#62606) creates one for each base table of an incrementally refreshed materialized view, named__doris_ivm_stream_<mv id>_<hash>.TableIf.TableType.toMysqlTypegives a stream the table typeVIEW.What problem does this PR solve?
Issue Number: None
Related PR: #61382, #62606
PR Type: Enhancement
Problem Summary: Metadata clients take a
VIEWrow for a view: they read its definition and parse it for lineage. A stream has no view definition;SHOW CREATE TABLEprintsCREATE STREAM ... ON TABLE .... OpenMetadata, for one, ingests every stream as a view whose definition is that statement, and the streams incremental refresh creates appear as views nobody created. This PR lists a stream asBASE TABLE: it is read with SELECT like a table, and MySQL defines no other table type that fits.ENGINEstaysStream, so clients can still tell streams apart. Table streams and incremental refresh are only on master, so no released version changes its output.Details
Before and after:
Table_typeof a stream inSHOW FULL TABLESVIEWBASE TABLETABLE_TYPE,ENGINEof a stream ininformation_schema.tablesVIEW,StreamBASE TABLE,Stream__doris_ivm_stream_*streams of an incrementally refreshed MVVIEWBASE TABLEViews keep
VIEWand system tablesSYSTEM VIEW.information_schema.views,SHOW STREAMS,SHOW CREATE TABLEandinformation_schema.table_streamsare unchanged: they select streams and views byTableType, not by this string.Configs and defaults: None.
Impact on existing users: None on released versions: table streams and incremental refresh are on neither branch-4.1 nor branch-4.2. On master,
SHOW FULL TABLES ... WHERE Table_type = 'VIEW'and JDBCgetTableswith the typeVIEWno longer return streams, and the typeTABLEnow does. Nothing is persisted: the type is computed whenever a listing is served.Testing
query_p0/schema_table/test_stream_table_typecreates a table, a stream on it and a view, and checks the stream and the view inSHOW FULL TABLES, all three ininformation_schema.tables, and thatinformation_schema.viewslists only the view. The.outis generated. On the code before this PR it fails at its first check:Expect cell is: BASE TABLE,But real is: VIEW.TableTypeTestchecks that a stream maps toBASE TABLEwith the engineStream, and thatVIEW,INLINE_VIEW,SCHEMA,OLAPandMATERIALIZED_VIEWkeep their types../run-fe-ut.sh --run org.apache.doris.catalog.TableTypeTest: 2 tests, 0 failures.__doris_ivm_stream_*stream is listed asBASE TABLE/Streamin both listings, and asVIEW/Streambefore this PR.6991e766329); the change is FE-only.Notes for Reviewers
BASE TABLEand not a new value such asSTREAM: MySQL definesBASE TABLE,VIEWandSYSTEM VIEW, and clients map and filter by them (MySQL Connector/J returnsBASE TABLErows asTABLEfromgetTables), so a new value would drop streams from client listings or break their type mapping.SHOW STREAMSandinformation_schema.table_streamsshow and that can be queried; hiding them by their name prefix in some listings would make the listings disagree.ENGINEininformation_schema.tablesis about to change ([fix](mtmv) Report MaterializedView as the ENGINE of async materialized views #70523), which the.outwould then depend on.Release note
None
Check List (For Author)
BASE TABLEinstead ofVIEW; table streams are not released yet.AI Disclosure