feat(agent): add Realtime voice sessions and a local Frankie demo - #7519
Draft
tlongwell-block wants to merge 10 commits into
Draft
feat(agent): add Realtime voice sessions and a local Frankie demo#7519tlongwell-block wants to merge 10 commits into
tlongwell-block wants to merge 10 commits into
Conversation
added 5 commits
September 8, 2026 22:11
The optional real-binary cancellation test otherwise waits for execution while its permission request is unanswered. Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Keep read and write halves independent so cancellation can be sent while provider output is pending. Add wire, input-bound, and disconnect tests. Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Keep a provider connection per ACP session and reuse the existing permission and tool executor. Support bounded PCM WAV input/output and reject unsupported policy modes. Exercise permission, conversation reuse, audio, and disconnect revocation through a real ACP subprocess. Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Explicitly request PCM24k on both audio directions. Accept absent or null manual turn detection and reject mismatched acknowledgment modalities. Return provider errors as transport events; the manual driver still fails closed on rejected mutations, now with a bounded provider diagnostic. Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Forward bounded ACP PNG/JPEG content as GA input_image parts alongside text. Advertise image input only for Realtime sessions and allow auto, text, or audio output through BUZZ_AGENT_REALTIME_OUTPUT. Validate input before provider mutation; cover forwarding, output choice, and invalid-image recovery with real ACP subprocess tests. Document the manual-turn limits and keep conventional text-only behavior unchanged. Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
🔐 Codex Security Review
|
tlongwell-block
marked this pull request as draft
September 9, 2026 14:11
added 3 commits
September 10, 2026 20:46
…e demo Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
* origin/main: fix(markdown): align mention chip wrapping (#7501) fix(relay): reject presence updates when Redis storage fails (#7532) fix(desktop): let inbox title and message author names truncate under narrow panes (#7550) fix(buzz-acp): report missing models without retrying (#7538) fix(desktop): require a Codex adapter with Astra support (#7427) fix(desktop): order unnamed roster members by full canonical npub (#7503) fix(mobile): standardize public-key identity display on npub (#7493) fix(desktop): npub identity controls across profile, agents, and workflows (#7489) fix(desktop): npub identity displays for mention, member, and workflow surfaces (#7495) fix(desktop): shared npub identity foundation (canonicalNpub, PubKey gate, strict parser) (#7488) fix(mobile): render push notification sender identity as npub (#7494) fix(acp): pace targeted overflow recovery on consumer capacity (#7325) fix(link-preview): keep composer fetches user-paced (#7211) Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
tlongwell-block
changed the title
added 2 commits
September 10, 2026 21:17
Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add OpenAI Realtime transport to
buzz-agent, including manual text/image/WAV turns and an opt-in ACP extension for live voice. A user can speak, interrupt playback, approve a shell command, and hear its actual result through the existing Buzz permission broker and MCP executor.The new
examples/realtime-audiowebpage streams native 24 kHz PCM, shows transcripts and playback latency, and lets users choose the brain's thinking level before connecting. A Tool approval menu defaults to Ask each time and can automatically approve subsequent shell calls; users can switch back while connected, and reload resets the choice. Its launcher starts the native llama.cpp Frankie server and Buzz with neutral instructions, an isolated default working directory, bounded lifetimes, and fresh loopback credentials. Users can supply their own voice WAV without rebuilding the GGUF.Capture, playback, provider responses, and tool execution have bounded queues and explicit cancellation/acknowledgment fences. Reconnect starts a fresh session without replaying tools. The example includes the sustained-capture, playback accounting, output-limit recovery, and transcript reset fixes exercised in native testing. Frankie's optional duplex/backchannel extensions are negotiated; the Realtime transport remains endpoint-agnostic.
No model weights, reference recordings, personal persona, machine paths, credentials, or raw session logs are included.
Related issue
Companion native implementation: tlongwell-block/llama.cpp#1
Try it
Follow the end-to-end setup guide. Obtain the model separately. The guide includes the tested GGUF's SHA-256, a pinned llama.cpp revision, Buzz build commands, and the launcher command. The page has no npm runtime dependencies or frontend build step. The launcher supports macOS and Linux.
Testing
cargo test -p buzz-agent: 721 passed, one pre-existing ignored test.node --test examples/realtime-audio/*.test.mjs: nine passed; production AudioWorklet/capture queues, auth/origin checks, permission forwarding, disconnect cleanup, and launcher rejection paths.node examples/realtime-audio/client.browser.mjs: production browser client and ACP adapter tested with a controlled subprocess for manual/automatic approval, switching modes, pending decisions, keyboard focus, reload defaults, and six viewport sizes.buzz-agentandbuzz-dev-mcp.d0aa7352535cbdc2bddfd5512b5f95d731761f72and GGUF SHA-2561822228c2caa051fb4fa8535c9a3105c2bb67440968dc5a9f6ee44ffbc439c50: spoken arithmetic, approveduptimewith a spoken result, a 1.5-second capture stall, reconnect with Minimal thinking, and denial without retrying. Desktop and narrow layouts checked. The two Off-thinking turns reached actual first sound in approximately 0.74 and 0.85 seconds, with no playback gaps in that run; these are fixture observations, not a general throughput benchmark.just ci: passed, including workspace/desktop Rust lint and tests, 6,488 desktop JavaScript tests, desktop/web builds, repository policy checks, and 2,098 mobile tests.Hosted OpenAI interoperability has not been live-verified. Native CPU/Metal compatibility has separate coverage; CUDA and a complete 128K workload on a 24 GiB consumer GPU remain hardware-validation work. The default 128K/Q4 KV allocation does not establish that GPU memory target.
New demo page (there was no webpage in the previous PR):