Skip to content

[improvement](stream) List table streams as BASE TABLE, not VIEW - #70560

Open
morningman wants to merge 1 commit into
apache:masterfrom
morningman:improve-stream-table-type
Open

morningman wants to merge 1 commit into
apache:masterfrom
morningman:improve-stream-table-type

Conversation

@morningman

Copy link
Copy Markdown
Contributor

Background

Since #61382, a table stream (CREATE STREAM s ON TABLE t) is listed with the tables and views of its database in SHOW FULL TABLES and information_schema.tables, with ENGINE Stream. Incremental refresh (#62606) creates one for each base table of an incrementally refreshed materialized view, named __doris_ivm_stream_<mv id>_<hash>. TableIf.TableType.toMysqlType gives a stream the table type VIEW.

What problem does this PR solve?

Issue Number: None

Related PR: #61382, #62606

PR Type: Enhancement

Problem Summary: Metadata clients take a VIEW row for a view: they read its definition and parse it for lineage. A stream has no view definition; SHOW CREATE TABLE prints CREATE STREAM ... ON TABLE .... OpenMetadata, for one, ingests every stream as a view whose definition is that statement, and the streams incremental refresh creates appear as views nobody created. This PR lists a stream as BASE TABLE: it is read with SELECT like a table, and MySQL defines no other table type that fits. ENGINE stays Stream, so clients can still tell streams apart. Table streams and incremental refresh are only on master, so no released version changes its output.

Details

  • Before and after:

    Before After
    Table_type of a stream in SHOW FULL TABLES VIEW BASE TABLE
    TABLE_TYPE, ENGINE of a stream in information_schema.tables VIEW, Stream BASE TABLE, Stream
    the __doris_ivm_stream_* streams of an incrementally refreshed MV listed, VIEW listed, BASE TABLE

    Views keep VIEW and system tables SYSTEM VIEW. information_schema.views, SHOW STREAMS, SHOW CREATE TABLE and information_schema.table_streams are unchanged: they select streams and views by TableType, not by this string.

  • Configs and defaults: None.

  • Impact on existing users: None on released versions: table streams and incremental refresh are on neither branch-4.1 nor branch-4.2. On master, SHOW FULL TABLES ... WHERE Table_type = 'VIEW' and JDBC getTables with the type VIEW no longer return streams, and the type TABLE now does. Nothing is persisted: the type is computed whenever a listing is served.

Testing

  • Regression test (new): query_p0/schema_table/test_stream_table_type creates a table, a stream on it and a view, and checks the stream and the view in SHOW FULL TABLES, all three in information_schema.tables, and that information_schema.views lists only the view. The .out is generated. On the code before this PR it fails at its first check: Expect cell is: BASE TABLE, But real is: VIEW.
  • FE unit test (new): TableTypeTest checks that a stream maps to BASE TABLE with the engine Stream, and that VIEW, INLINE_VIEW, SCHEMA, OLAP and MATERIALIZED_VIEW keep their types. ./run-fe-ut.sh --run org.apache.doris.catalog.TableTypeTest: 2 tests, 0 failures.
  • Manual: an incrementally refreshed materialized view over a merge-on-write table with changelog; its __doris_ivm_stream_* stream is listed as BASE TABLE / Stream in both listings, and as VIEW / Stream before this PR.
  • Environment: a local cluster whose FE ran this branch's fe-core jar (and, for the before runs, a jar of the same base without this change), with a BE from an earlier master build (6991e766329); the change is FE-only.

Notes for Reviewers

  • Why BASE TABLE and not a new value such as STREAM: MySQL defines BASE TABLE, VIEW and SYSTEM VIEW, and clients map and filter by them (MySQL Connector/J returns BASE TABLE rows as TABLE from getTables), so a new value would drop streams from client listings or break their type mapping.
  • Why the streams incremental refresh creates stay listed: they are real objects that SHOW STREAMS and information_schema.table_streams show and that can be queried; hiding them by their name prefix in some listings would make the listings disagree.
  • The incremental-refresh case is checked by hand rather than in the suite: it needs a merge-on-write base table with changelog, and a materialized view's ENGINE in information_schema.tables is about to change ([fix](mtmv) Report MaterializedView as the ENGINE of async materialized views #70523), which the .out would then depend on.

Release note

None

Check List (For Author)

  • Behavior changed: Yes. Table streams are listed as BASE TABLE instead of VIEW; table streams are not released yet.
  • Does this need documentation: No. doris-website has no page on table streams yet.
  • Pre-submission status: Ready

AI Disclosure

  • Tool: Claude Code (Claude Opus 5.5)
  • Generated: the code change, both tests and this description
  • Human review: not stated
  • Prompts: While integrating OpenMetadata with Doris, asked to fix in Doris that table streams are listed as views; chose to list every stream as BASE TABLE and not to hide the streams that incremental refresh creates.

### Background

Since apache#61382, a table stream (`CREATE STREAM s ON TABLE t`) is listed with the tables and views of its database in `SHOW FULL TABLES` and `information_schema.tables`, with `ENGINE` `Stream`. Incremental refresh (apache#62606) creates one for each base table of an incrementally refreshed materialized view, named `__doris_ivm_stream_<mv id>_<hash>`. `TableIf.TableType.toMysqlType` gives a stream the table type `VIEW`.

### What problem does this PR solve?

Issue Number: None

Related PR: apache#61382, apache#62606

PR Type: Enhancement

Problem Summary: Metadata clients take a `VIEW` row for a view: they read its definition and parse it for lineage. A stream has no view definition; `SHOW CREATE TABLE` prints `CREATE STREAM ... ON TABLE ...`. OpenMetadata, for one, ingests every stream as a view whose definition is that statement, and the streams incremental refresh creates appear as views nobody created. This PR lists a stream as `BASE TABLE`: it is read with SELECT like a table, and MySQL defines no other table type that fits. `ENGINE` stays `Stream`, so clients can still tell streams apart. Table streams and incremental refresh are only on master, so no released version changes its output.

### Details

- **Before and after:**

  | | Before | After |
  |---|---|---|
  | `Table_type` of a stream in `SHOW FULL TABLES` | `VIEW` | `BASE TABLE` |
  | `TABLE_TYPE`, `ENGINE` of a stream in `information_schema.tables` | `VIEW`, `Stream` | `BASE TABLE`, `Stream` |
  | the `__doris_ivm_stream_*` streams of an incrementally refreshed MV | listed, `VIEW` | listed, `BASE TABLE` |

  Views keep `VIEW` and system tables `SYSTEM VIEW`. `information_schema.views`, `SHOW STREAMS`, `SHOW CREATE TABLE` and `information_schema.table_streams` are unchanged: they select streams and views by `TableType`, not by this string.
- **Configs and defaults:** None.
- **Impact on existing users:** None on released versions: table streams and incremental refresh are on neither branch-4.1 nor branch-4.2. On master, `SHOW FULL TABLES ... WHERE Table_type = 'VIEW'` and JDBC `getTables` with the type `VIEW` no longer return streams, and the type `TABLE` now does. Nothing is persisted: the type is computed whenever a listing is served.

### Testing

- Regression test (new): `query_p0/schema_table/test_stream_table_type` creates a table, a stream on it and a view, and checks the stream and the view in `SHOW FULL TABLES`, all three in `information_schema.tables`, and that `information_schema.views` lists only the view. The `.out` is generated. On the code before this PR it fails at its first check: `Expect cell is: BASE TABLE`, `But real is: VIEW`.
- FE unit test (new): `TableTypeTest` checks that a stream maps to `BASE TABLE` with the engine `Stream`, and that `VIEW`, `INLINE_VIEW`, `SCHEMA`, `OLAP` and `MATERIALIZED_VIEW` keep their types. `./run-fe-ut.sh --run org.apache.doris.catalog.TableTypeTest`: 2 tests, 0 failures.
- Manual: an incrementally refreshed materialized view over a merge-on-write table with changelog; its `__doris_ivm_stream_*` stream is listed as `BASE TABLE` / `Stream` in both listings, and as `VIEW` / `Stream` before this PR.
- Environment: a local cluster whose FE ran this branch's fe-core jar (and, for the before runs, a jar of the same base without this change), with a BE from an earlier master build (`6991e766329`); the change is FE-only.

### Notes for Reviewers

- Why `BASE TABLE` and not a new value such as `STREAM`: MySQL defines `BASE TABLE`, `VIEW` and `SYSTEM VIEW`, and clients map and filter by them (MySQL Connector/J returns `BASE TABLE` rows as `TABLE` from `getTables`), so a new value would drop streams from client listings or break their type mapping.
- Why the streams incremental refresh creates stay listed: they are real objects that `SHOW STREAMS` and `information_schema.table_streams` show and that can be queried; hiding them by their name prefix in some listings would make the listings disagree.
- The incremental-refresh case is checked by hand rather than in the suite: it needs a merge-on-write base table with changelog, and a materialized view's `ENGINE` in `information_schema.tables` is about to change (apache#70523), which the `.out` would then depend on.

### Release note

None

### Check List (For Author)

- Behavior changed: Yes. Table streams are listed as `BASE TABLE` instead of `VIEW`; table streams are not released yet.
- Does this need documentation: No. doris-website has no page on table streams yet.
- Pre-submission status: Ready

### AI Disclosure

- Tool: Claude Code (Claude Opus 5.5)
- Generated: the code change, both tests and this description
- Human review: not stated

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant