Skip to content

Add streaming query results - #395

Draft
jievince wants to merge 8 commits into
vesoft-inc:devfrom
jievince:response-stream
Draft

jievince wants to merge 8 commits into
vesoft-inc:devfrom
jievince:response-stream

Conversation

@jievince

Copy link
Copy Markdown
Contributor

graphd adds a StreamingExecute RPC that returns a query result as a stream of messages: one header (metadata), any number of batch messages, and one final carrying the status, summary and cursor (https://github.com/vesoft-inc/nebula-ng/pull/11432). Execute still returns the whole result in one message, so the client must receive and decode it completely before the first row is readable.

This PR adds forward-only streaming results to the Go client without changing types.Client or types.Result, so existing application mocks keep compiling:

type StreamingClient interface {
    StreamingExecute(ctx context.Context, stmt string) (StreamingResult, error)
}

type StreamingResult interface {
    types.Result   // existing row-reading functions accept it
    Err() error    // check after HasNext returns false
    Close() error  // abandon the stream early
}

The clients returned by the SDK implement StreamingClient; callers obtain it through a type assertion.

Reading model

  • StreamingExecute returns after the first response message, so statement errors and column names are available at the call site, as with Execute.
  • The next message is received only after the current batch has been read. This keeps the SDK from buffering the result and lets a slow consumer push back on the server through gRPC flow control.
  • Batches are decoded with the existing unary vector decoder.
  • A result succeeds only with a successful final followed by a normal end of stream. A missing final, a message after final, a header with result totals, or an empty batch is reported through Err().
  • RowSize() returns -1: the total is unknown while streaming.

Connection rules

  • One connection has at most one active stream. Until it is read or closed, Execute, Ping and another StreamingExecute on that client return an error, and the pool refuses to take the client back.
  • Close() cancels only this RPC and returns without waiting for the server to stop the query. It may be called more than once and from another goroutine to interrupt a blocked read.
  • StreamingExecute uses only the caller's ctx; its deadline covers the whole iteration, including the time spent processing rows. A transport error is not retried, because the statement may already have run.

Released graphd versions declare StreamingExecute but do not handle it, so calling them waits instead of failing with UNIMPLEMENTED; there is no version negotiation. Usage is documented in docs/streaming.md.

Only graph.pb.go and graph_grpc.pb.go are regenerated for the new messages; vector.proto is unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant