[italocjs]

Fixes #287

What and why

Tool calls are buffered until Kiro finishes the whole response, so while the model writes a large tool input (e.g. Claude Code writing a big file) the gateway sends the client no bytes at all, sometimes for minutes. Clients with an idle-stream watchdog treat that as a dead connection. Claude Code aborts with "Waiting for API response … check your network" and retries the entire turn. The real Anthropic API never goes silent like this: it sends ping events and streams tool input.

This PR adds a keepalive that only fires during silence:

• streaming_core.with_keepalive(source, keepalive_chunk, interval): passes chunks through unchanged and inserts keepalive_chunk after every interval seconds without a chunk.
  • Keepalives start only after the first real chunk, so HTTP errors and first-token-timeout errors raised before the stream starts behave exactly as before.
  • On client disconnect it cancels the pending read and closes the source, so the upstream response is still released.
• Anthropic (stream_with_first_token_retry_anthropic): sends event: ping / {"type": "ping"}, the API's own keepalive, which SDKs skip.
• OpenAI (stream_with_first_token_retry): sends : keepalive, an SSE comment line that clients ignore per the SSE spec.
• Config: STREAM_KEEPALIVE_INTERVAL, default 10 seconds; 0 disables it. Documented in config.py and .env.example.
• Non-streaming mode is unaffected (no SSE). Both route paths that stream, including the web-search one, go through the two wrapped functions.

Test coverage

15 new tests, full suite green (1706 passed):

• TestWithKeepalive (core, 9 tests):
  • fast stream passes through unchanged
  • keepalives are inserted during silence
  • no keepalive before the first chunk
  • interval=0 disables keepalives
  • an empty source finishes cleanly
  • an error before the first chunk propagates
  • an error mid-stream propagates
  • a client disconnect closes the source and cancels the pending read
  • chunk order is preserved
• TestStreamingAnthropicKeepalive: ping events sent while a tool call is buffered; none when disabled.
• TestStreamingOpenaiKeepalive: SSE comments sent while a tool call is buffered; none when disabled.
• TestStreamKeepaliveIntervalConfig: default value and env override.

I checked that the Anthropic keepalive test fails on main (assert len(pings) >= 2).

Real-world verification

Running in my local Docker deployment:

• Raw request with a forced ~21 KB tool call: before the fix, nothing arrived between message_start (3.6 s) and the tool block (43 s). After it, ping arrives at 13.6 s, 23.6 s and 33.6 s.
• Claude Code (2.1.304) through the gateway with CLAUDE_BYTE_STREAM_IDLE_TIMEOUT_MS=15000, asked to Write a ~400-line file: the ~54 s stream finished with no retry and wrote all 662 lines. Without the fix, that watchdog aborts the request.

[cla-bot]

Thanks for the PR! 🎉

Before merge, we need a one-time CLA confirmation.
It confirms that you have the right to contribute this code and allow the project to use it.

Full CLA text:
https://github.com/jwadow/kiro-gateway/blob/main/CLA.md

Please reply once with:

I have read the CLA and I accept its terms

You need to write once, all further messages from me can be ignored.

[italocjs]

I have read the CLA and I accept its terms

I have read the CLA and I accept its terms