Integrate tool calling with Engine chat batching - #1082
Draft
David Fan (jiafatom) wants to merge 11 commits into
Draft
Integrate tool calling with Engine chat batching#1082David Fan (jiafatom) wants to merge 11 commits into
David Fan (jiafatom) wants to merge 11 commits into
Conversation
Use each model's genai_config batching contract so independent chat sessions can share an ORT GenAI Engine while Generator models retain their existing behavior. Files changed: - Add the owner-thread Engine dispatcher and per-session adapter. - Route chat creation, continuation, cancellation, usage, and option changes through the selected backend. - Parse and validate static and dynamic Engine configuration. - Add configuration, template, concurrency, and search-option coverage. - Pin ORT GenAI and packaging to 0.15.3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9b90e66e-af55-404d-b606-7899f6f73c54
Remove compatibility branches for OGA releases without Engine support so missing APIs fail during compilation instead of at model load. Files changed: - sdk_v2/cpp/CMakeLists.txt - sdk_v2/cpp/src/inferencing/generative/chat/chat_session.cc - sdk_v2/cpp/src/inferencing/generative/genai_model_instance.cc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9b90e66e-af55-404d-b606-7899f6f73c54
Retain OGA 0.15.2 until the next Engine-capable stable package is published, while non-release CI continues using the selected nightly. Remove optional OGA hooks from the shared test entry point because not every test target consumes OGA. Files changed: - .pipelines/foundry-local-packaging.yml - .pipelines/v2/sdk_v2-pipeline-plan.md - sdk_v2/deps_versions.json - sdk_v2/cpp/test/test_main.cc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9b90e66e-af55-404d-b606-7899f6f73c54
The shared test entry point is compiled by targets with and without OGA include paths. Retain header detection so OGA-linked tests shut down cleanly without imposing that dependency on cache-only tests. Files changed: - sdk_v2/cpp/test/test_main.cc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9b90e66e-af55-404d-b606-7899f6f73c54
Destroy retained generators before releasing model ownership, preserve undo correctness after generator rebuilds, and keep the Generator path's implicit limit behavior unchanged. Use bounded Generator context for one-shot OpenAI JSON requests and map the CI nightly's native StopSequence finish reason. Stage the shared chat model with dynamic Engine batching so concurrency and cancellation coverage exercise the dispatcher. Files changed: chat_session.cc, chat_session.h, onnx_engine_chat_generator.cc, chat_session_test.cc. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Invalidate retained requests when explicit guidance changes, and close newly-created Engine conversations if turn setup or adapter construction fails. Accept both StopSequence and StopString finish-reason symbols while compiling against the exact non-release CI nightly. Files changed: chat_session.cc, onnx_engine_chat_generator.cc. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Decode model-configured BOT/EOT/BOR/EOR token IDs exactly like the Generator backend instead of matching English token spellings. Stage a static-batching variant of the shared model and verify that a rebuilt second turn retains committed conversation history. Files changed: onnx_engine_chat_generator.cc/.h, chat_session_test.cc. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Evict dormant dynamic conversations under capacity pressure so blocked turns can progress and rebuild evicted state from committed chat history. Stage the shared CPU model in a writable temp directory and exercise only its supported static Engine configuration. Files changed: chat_session.cc, onnx_chat_engine.cc, onnx_chat_engine.h, chat_session_test.cc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Only completed conversations with a nonzero turn ID are dormant candidates, preventing capacity pressure from closing a request between creation and its first BeginTurn. Files changed: onnx_chat_engine.cc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
David Fan (jiafatom)
marked this pull request as draft
September 8, 2026 02:53
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Final response construction, reasoning output, tool-call continuity, Engine finish reasons, and catalog persistence contain blocking regressions.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Integrates model-selected ORT GenAI Engine batching with existing C++ chat, tool-calling, reasoning, and local-model support.
Changes:
- Adds static/dynamic Engine-backed chat generation and configuration.
- Expands tool-call parsing, history serialization, and Responses API handling.
- Updates reasoning, streaming, catalog, and integration tests.
File summaries
| File | Description |
|---|---|
sdk_v2/cpp/test/test_main.cc |
Adds OGA shutdown handling. |
sdk_v2/cpp/test/internal_api/toolcalling/tool_call_utils_test.cc |
Tests malformed-call recovery and aliases. |
sdk_v2/cpp/test/internal_api/toolcalling/tool_call_stream_accumulator_test.cc |
Updates accumulator output tests. |
sdk_v2/cpp/test/internal_api/toolcalling/grammar_test.cc |
Updates generated grammar expectations. |
sdk_v2/cpp/test/internal_api/response_converter_test.cc |
Tests function-call input conversion. |
sdk_v2/cpp/test/internal_api/genai_config_test.cc |
Tests Engine configuration parsing. |
sdk_v2/cpp/test/internal_api/chat/search_options_test.cc |
Tests limits and retained settings. |
sdk_v2/cpp/test/internal_api/chat/chat_template_test.cc |
Tests tool history and continuation prompts. |
sdk_v2/cpp/test/internal_api/chat/chat_session_test.cc |
Adds static Engine integration coverage. |
sdk_v2/cpp/src/service/responses_handler.cc |
Normalizes tool names and streaming events. |
sdk_v2/cpp/src/items/message_item.h |
Stores assistant tool calls. |
sdk_v2/cpp/src/items/message_item.cc |
Copies stored tool calls. |
sdk_v2/cpp/src/inferencing/generative/toolcalling/tool_call_utils.h |
Extends parser inputs. |
sdk_v2/cpp/src/inferencing/generative/toolcalling/tool_call_utils.cc |
Adds malformed-output recovery and aliases. |
sdk_v2/cpp/src/inferencing/generative/toolcalling/tool_call_stream_accumulator.h |
Adds visible text and ready-call outputs. |
sdk_v2/cpp/src/inferencing/generative/toolcalling/grammar.cc |
Constrains mixed text/tool grammar. |
sdk_v2/cpp/src/inferencing/generative/openresponses/response_converter.cc |
Converts function-call inputs. |
sdk_v2/cpp/src/inferencing/generative/genai_model_instance.h |
Owns the shared chat Engine. |
sdk_v2/cpp/src/inferencing/generative/genai_model_instance.cc |
Initializes and validates Engine use. |
sdk_v2/cpp/src/inferencing/generative/genai_config.h |
Defines Engine configuration types. |
sdk_v2/cpp/src/inferencing/generative/genai_config.cc |
Parses and validates Engine settings. |
sdk_v2/cpp/src/inferencing/generative/chat/search_options.h |
Adds retained-setting and guidance APIs. |
sdk_v2/cpp/src/inferencing/generative/chat/search_options.cc |
Applies limits and grammar guidance. |
sdk_v2/cpp/src/inferencing/generative/chat/reasoning_stream_splitter.h |
Supports prefilled reasoning state. |
sdk_v2/cpp/src/inferencing/generative/chat/onnx_engine_chat_generator.h |
Defines the Engine generator adapter. |
sdk_v2/cpp/src/inferencing/generative/chat/onnx_engine_chat_generator.cc |
Implements Engine-backed generation. |
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_generator.h |
Extends Generator state and decoding. |
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_generator.cc |
Adds special-token and reasoning handling. |
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_engine.h |
Defines the Engine dispatcher. |
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_engine.cc |
Implements scheduling and event routing. |
sdk_v2/cpp/src/inferencing/generative/chat/chat_template.h |
Exposes message and continuation serialization. |
sdk_v2/cpp/src/inferencing/generative/chat/chat_template.cc |
Serializes tool calls and boundaries. |
sdk_v2/cpp/src/inferencing/generative/chat/chat_session.h |
Generalizes retained chat backends. |
sdk_v2/cpp/src/inferencing/generative/chat/chat_session.cc |
Integrates Engine, tools, and reasoning. |
sdk_v2/cpp/src/inferencing/generative/chat/chat_generator.h |
Expands the generator abstraction. |
sdk_v2/cpp/src/inferencing/generative/chat/chat_generator.cc |
Adds optional turn usage. |
sdk_v2/cpp/src/contracts/responses.h |
Adds function-call input items. |
sdk_v2/cpp/src/contracts/responses_json.cc |
Parses function-call inputs. |
sdk_v2/cpp/src/catalog/azure_model_catalog.h |
Revises local-model merging. |
sdk_v2/cpp/src/catalog/azure_model_catalog.cc |
Adds BYOM metadata preservation. |
sdk_v2/cpp/CMakeLists.txt |
Builds the Engine implementation. |
docs/README.md |
Links tool-calling engineering notes. |
Review details
- Files reviewed: 42/42 changed files
- Comments generated: 17
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+886
to
889
| // REASONING segments: intentionally dropped from the Chat Completions stream. Never feed reasoning text to | ||
| // the tool-call accumulator — tool-call-shaped text inside <think>...</think> is scratchpad, not a real call. | ||
| if (seg.type != FOUNDRY_LOCAL_TEXT_ITEM_TYPE_DEFAULT) { | ||
| if (!tool_accumulator.InsideToolCall()) { | ||
| process_tool_output(tool_accumulator.Flush()); | ||
| } | ||
| AppendGeneratedSegment(generated_events, seg.text, seg.type); | ||
|
|
||
| if (is_streaming && !seg.text.empty()) { | ||
| auto chunk_json = chat_completions::FormatReasoningStreamingChunk( | ||
| seg.text, completion_id, created, model_name); | ||
| streaming_callback->PushItem(std::make_unique<TextItem>( | ||
| std::move(chunk_json), FOUNDRY_LOCAL_TEXT_ITEM_TYPE_OPENAI_JSON)); | ||
| } | ||
| continue; |
Comment on lines
+198
to
+199
| model_infos.push_back(MakeByomModelInfo(model_id, local_path)); | ||
| models.push_back(model_factory_(ModelInfo(model_infos.back()), local_path)); |
Comment on lines
+241
to
+247
| // Locally imported models may not have catalog metadata even when their | ||
| // chat template uses the standard Qwen tool-call markers. | ||
| if (tool_ctx.HasTools() && tool_ctx.tool_call_start.empty() && tool_ctx.tool_call_end.empty()) { | ||
| tool_ctx.supports_tool_calling = true; | ||
| tool_ctx.tool_call_start = "<tool_call>"; | ||
| tool_ctx.tool_call_end = "</tool_call>"; | ||
| } |
Comment on lines
+365
to
+369
| if (has_tool_calls) { | ||
| // Add structured tool call items to the response | ||
| auto tool_items = ToolCallsToItems(parsed_calls); | ||
| for (auto& ti : tool_items) { | ||
| response.items.push_back(std::move(ti)); |
Comment on lines
+378
to
+381
| if (tool_ctx.supports_reasoning) { | ||
| std::string start = tool_ctx.reasoning_start.empty() ? "<think>" : tool_ctx.reasoning_start; | ||
| std::string end = tool_ctx.reasoning_end.empty() ? "</think>" : tool_ctx.reasoning_end; | ||
| segments = SplitReasoningContent(text, start, end); |
Comment on lines
+331
to
+333
| } else if (auto* fc = std::get_if<FunctionCallInputItem>(&input_item)) { | ||
| auto i = std::make_unique<ToolCallItem>(fc->call_id, fc->name, fc->arguments); | ||
| request.AddOwnedItem(std::move(i)); |
| } | ||
|
|
||
| loaded = ctx_.model_load_manager.GetLoadedModel(model->Id(), model->GetPath()); | ||
| loaded = ctx_.model_load_manager.GetLoadedModel(model->Id()); |
|
|
||
| Documentation for Foundry Local can be found in the following resources: | ||
|
|
||
| - [MSBench Tool Calling Fixes](MSBench%20Tool%20Calling%20Fixes.md): Engineering notes for the Responses API and tool-calling changes that enabled a local Qwen model to produce SWE-bench patches through MSBench. |
Comment on lines
87
to
+94
| auto model_path = fl::test::GetTestModelPath(fl::test::kTestChatModelAlias); | ||
| engine_model_ = std::make_unique<EngineModelStaging>(model_path); | ||
| logger_ = std::make_unique<StderrLogger>(); | ||
| ep_detector_ = std::make_unique<test::CpuOnlyEpDetector>(); | ||
| load_manager_ = std::make_unique<ModelLoadManager>(*ep_detector_, *logger_); | ||
|
|
||
| auto result = load_manager_->LoadModel( | ||
| model_path.string(), | ||
| engine_model_->path().string(), |
Comment on lines
+4
to
+5
| #include <gtest/gtest.h> | ||
| #include <gtest/gtest.h> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
cmake --build sdk_v2/cpp/build/Windows/Release --config Release --target foundry_local_testsFOUNDRY_TEST_DATA_DIRwas not set; no individual test failuresThis branch preserves the commit order from #1060 followed by #1071, with a final integration commit for conflicts exposed by compilation and focused tests.