Multimodal content blocks

[Zhuoxi2000]

Linked issue: #1059 (Phase 1 — the tracking issue stays open for the provider phase)

Purpose of change

Implements the framework part of the multimodal design agreed in #1031.

ChatMessage.content: String is replaced by blocks: List<ContentBlock> in both Java and Python. The initial block types are TextBlock, ImageBlock, AudioBlock, VideoBlock, and DocumentBlock. Media blocks share a MIME-typed shape with mime_type, exactly one of base64 data or external url, and optional metadata such as name, size_bytes, and sha256.

Existing text constructors/factories keep their signatures and create a single TextBlock. getText() / .text returns the ordered text projection.

Java and Python use the same type-discriminated JSON format, covered by the regenerated cross-language snapshots. The pemja bridge is updated to carry blocks across runtimes as well.

This PR also updates prompt handling so image-only messages are preserved, and Prompt.formatMessages applies substitutions only to text blocks while passing media blocks through unchanged.

Provider-specific multimodal conversion is intentionally left to the next #1059 phase. Existing providers continue using the text projection in this PR.

Behavior changes:

• CEL expressions using response.content need to migrate to response.blocks.
• Python ChatMessage now rejects unknown fields, so the removed content= argument fails explicitly instead of silently creating an empty message.
• toString() / __str__, equality, and hashing now use the block representation.
• data and url are mutually exclusive for media blocks.

Tests

• Added Java and Python ChatMessage serialization tests covering the wire format, mixed block ordering, media fields, round trips, source validation, and rejection of the removed content argument.
• Regenerated the Java/Python cross-language snapshots using the existing snapshot flows; cross-deserialization passes in both directions.
• Added durable-state serde coverage for a mixed text + image ChatMessage.
• Java reactor tests and Python tests pass locally, aside from environment-specific dependency failures that are covered by CI.

API

Yes. This is a Beta breaking change discussed in #1031.

ChatMessage.content is replaced by blocks in both Java and Python. Java removes getContent()/setContent; Python removes .content. Text-only access remains available through getText() / .text and setText() / set_text().

New public types in both languages are ContentBlock, TextBlock, MediaBlock, ImageBlock, AudioBlock, VideoBlock, and DocumentBlock.

ChatMessage also adds getBlocksAsMaps() / setBlocksFromMaps() for the pemja bridge, where blocks cross as plain maps.

CEL expressions referencing response.content must migrate to response.blocks.

Documentation

• doc-needed
• doc-not-needed
• doc-included

Was this patch authored or co-authored using generative AI tooling?

• Yes
• No

Generated by: Claude Code Fable 5

[wenjin272]

Thanks for taking this on. Apologies for the delayed review due to our recent team outing. I will complete the review by the end of this week.

[wenjin272]

Thanks for the work on this comprehensive multimodal message refactor! I left a few comments on the Event Log serialization contract, cross-language migration and invariants, routing isolation, and documentation updates.

[wenjin272]

We need an explicit serialization contract for Event Log output here. The regular ChatMessage JSON representation is also used by the Java/Python bridge, event serialization, and state recovery, where data and the original url must be preserved. Therefore, changing the global ChatMessage serialization to remove these fields would break the normal wire format. Reusing that same representation for Event Log output, however, cannot guarantee metadata-only and sanitized logging.

Could each ContentBlock define its log-safe representation—for example through sanitize()—and could the Event Log serialization module register a dedicated ChatMessage serializer that uses it? A media block should explicitly whitelist its loggable metadata, omit inline data, and sanitize credentials from URLs. This keeps EventLogRecordJsonSerializer generic, avoids hard-coding every block subtype into it, and lets future block types define their own logging policy. Tests should verify both that normal JSON serialization still preserves the complete payload and that Event Log serialization never exposes it at either STANDARD or VERBOSE level.

[wenjin272]

This test still constructs the serialized Python messages with the old content field, while the production Python resource now serializes them as blocks. PythonPrompt.parseChatMessage() still reads content and defaults it to an empty string, so a real message-based Python prompt loses all of its blocks when restored in Java. Could we parse blocks through the existing map/Jackson conversion and update this fixture to use the actual Python model_dump() shape, including text and media blocks?

[wenjin272]

This copies the block list but still shares every mutable ContentBlock instance with the original request. For example, a strategy can obtain the first TextBlock from context.getMessages(), call setText("mutated"), and the original message later sent to the model will also return "mutated". The same aliasing exists for MediaBlock.setData() and setUrl(). This does not require a hostile strategy; it is exactly the kind of accidental mutation that the defensive copy is intended to isolate, and it is newly exposed because the previous text content was an immutable String.

Rather than adding a type-specific deep-copy branch whenever a new block type is introduced, could we make the concrete ContentBlock implementations immutable—final fields, no setters, and validated @JsonCreator constructors—and snapshot the block list in ChatMessage? RoutingContext could then safely share block instances while continuing to copy the mutable message/list structure. The routing isolation tests should also cover block-level aliasing rather than only replacing the copied message's text/list.

[wenjin272]

The constructor enforces exactly one of data and url, but the no-arg Jackson path and the independent setters bypass that check. Java can therefore accept a wire payload containing both or neither source while Python rejects the same payload. Could we use validated immutable @JsonCreator construction—or represent the source as Base64Source | URLSource—and freeze or validate assignment on the Python side as well?

[wenjin272]

This response access was migrated, but the same documentation still contains ChatMessage(..., content=...) examples and Java calls to the removed getContent() method. The Python examples now fail because unknown fields are forbidden, while the Java examples no longer compile. Similar remnants remain in prompts.md, react_agent.md, workflow_agent.md, and the quickstarts. Could we complete a repository-wide documentation migration to blocks/the message factories and text/getText()?

[wenjin272]

Would mediaType be a better name than mimeType here? Media Type is the terminology used by the current standards, while MIME type is more of a commonly used historical name. Since this field is part of the public multimodal ContentBlock API rather than a MIME-specific protocol implementation, mediaType may better describe its semantics and also reads naturally alongside MediaBlock.

This is mostly an API naming suggestion, but since these block types are being introduced in this PR, it may be worth settling the terminology before the API becomes public.