Add typed VideoURL multimodal support - #276
Merged
Merged
Conversation
Providers like MiniMax accept OpenAI-compatible video_url content parts, but Instructor has no video type (567-labs/instructor#2520). VideoURL fills the gap: ChatHistory extracts it like Instructor's Image/Audio/PDF and get_history() emits its content part, which Instructor forwards to the provider unchanged. Token counting turns video parts into text placeholders because LiteLLM's token_counter raises on video_url parts. Supersedes the raw-dict approach from #274. Closes #275 Co-Authored-By: octo-patch <266937838+octo-patch@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Map docs updated by the update-codebase-map pass for HEAD c80c752: VideoURL public export, ChatHistory video_url serialization, and the LiteLLM token-count placeholder are now reflected in the affected docs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- to_openai() uses model_dump(exclude_none=True) instead of hand-rolled optional-field assembly - Single MULTIMODAL_TYPES registry; drop the single-use type aliases - Token-count serializer handles any pre-lowered content-part dict with a typed placeholder instead of matching the "video_url" literal - Replace the unreachable exclude-spec guard with an invariant assert - Video test asserts the serialized shape only; drop the unmocked litellm call (tiktoken fetches BPE files over the network on cold caches) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Typed video input support, superseding the approach in #274 (thanks @octo-patch for kicking this off and for the MiniMax use case).
Instructor has no video type (asked upstream: 567-labs/instructor#2520), so this adds our own
VideoURL:ChatHistorytreatsVideoURLas a first-class multimodal type;get_history()emits it as an OpenAI-compatiblevideo_urlcontent part, which Instructor forwards to the provider unchanged[video content]text placeholder. LiteLLM'stoken_counterraises onvideo_urlparts, which is what broke fix: preserve video content parts in chat history #274:get_context_token_count()crashed whenevermax_context_tokenswas setdump()/load()as a plain Pydantic modelCloses #275. When Instructor grows a native
Videotype we can swap it in behind the same seam.Checks:
uv run black(unchanged),uv run flake8(clean),uv run pytest --cov=atomic_agents atomic-agents(333 passed;test_get_class_from_stringalso fails locally for me on clean main, pre-existing rootdir quirk, unrelated).🤖 Generated with Claude Code