Skip to content

Add typed VideoURL multimodal support - #276

Merged
Eigenwise merged 3 commits into
mainfrom
feat/video-url
Aug 11, 2026
Merged

Add typed VideoURL multimodal support#276
Eigenwise merged 3 commits into
mainfrom
feat/video-url

Conversation

@Eigenwise

Copy link
Copy Markdown
Owner

Typed video input support, superseding the approach in #274 (thanks @octo-patch for kicking this off and for the MiniMax use case).

Instructor has no video type (asked upstream: 567-labs/instructor#2520), so this adds our own VideoURL:

from atomic_agents import BaseIOSchema, VideoURL
from pydantic import Field

class VideoAnalysisInput(BaseIOSchema):
    """Input with a video for analysis"""
    question: str = Field(..., description="Question about the video")
    video: VideoURL = Field(..., description="Video to analyze")
  • ChatHistory treats VideoURL as a first-class multimodal type; get_history() emits it as an OpenAI-compatible video_url content part, which Instructor forwards to the provider unchanged
  • Token counting turns video parts into a [video content] text placeholder. LiteLLM's token_counter raises on video_url parts, which is what broke fix: preserve video content parts in chat history #274: get_context_token_count() crashed whenever max_context_tokens was set
  • Round-trips through dump() / load() as a plain Pydantic model
  • Docs section in the memory guide

Closes #275. When Instructor grows a native Video type we can swap it in behind the same seam.

Checks: uv run black (unchanged), uv run flake8 (clean), uv run pytest --cov=atomic_agents atomic-agents (333 passed; test_get_class_from_string also fails locally for me on clean main, pre-existing rootdir quirk, unrelated).

🤖 Generated with Claude Code

Providers like MiniMax accept OpenAI-compatible video_url content parts,
but Instructor has no video type (567-labs/instructor#2520). VideoURL
fills the gap: ChatHistory extracts it like Instructor's Image/Audio/PDF
and get_history() emits its content part, which Instructor forwards to
the provider unchanged. Token counting turns video parts into text
placeholders because LiteLLM's token_counter raises on video_url parts.

Supersedes the raw-dict approach from #274.

Closes #275

Co-Authored-By: octo-patch <266937838+octo-patch@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eigenwise and others added 2 commits August 11, 2026 13:04
Map docs updated by the update-codebase-map pass for HEAD c80c752:
VideoURL public export, ChatHistory video_url serialization, and the
LiteLLM token-count placeholder are now reflected in the affected docs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- to_openai() uses model_dump(exclude_none=True) instead of hand-rolled
  optional-field assembly
- Single MULTIMODAL_TYPES registry; drop the single-use type aliases
- Token-count serializer handles any pre-lowered content-part dict with a
  typed placeholder instead of matching the "video_url" literal
- Replace the unreachable exclude-spec guard with an invariant assert
- Video test asserts the serialized shape only; drop the unmocked litellm
  call (tiktoken fetches BPE files over the network on cold caches)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Eigenwise
Eigenwise merged commit a32edaf into main Aug 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Typed video input support: VideoURL multimodal class

1 participant