[api][java][python] Align multimodal message contracts and safe logging

[wenjin272]

Linked issue: #1059 (follow-up to #1060; the tracking issue remains open)

Purpose of change

Preserve multimodal responses during Java structured-output parsing, prevent raw media from appearing in the Python chat failure debug message, and align Java/Python media field validation.

Runtime flow

Java structured-output parsing reads the text projection, parses it into the requested schema, and returns a message preserving the original blocks and tool calls with the parsed value added to extraArgs. Python's terminal FAIL path logs the request ID and message count, then rethrows the original exception. Media construction/deserialization validates field types before messages enter normal JSON, bridge, or state paths.

Key decisions

• Apply strict media-field validation locally in Jackson to align with Python without changing unrelated deserialization behavior.
• Preserve the original response when attaching structured output; text cleanup is used only for parsing.
• Keep diagnostic representations free of raw media sources while preserving complete payloads in normal serialization.

Behavioral Semantics

Interaction decisions

Conditions	Result
Structured output + text/media blocks	Parse the text projection; preserve all original blocks, text formatting, tool calls, and existing metadata.
Python chat failure + FAIL strategy	Log request ID/count without rendering input messages; propagate the exception. IGNORE/RETRY behavior is unchanged.
Media + ordinary JSON or bridge maps	Preserve payload/URL; apply the same strict media-field rules during deserialization.
Media + diagnostic representation	Python source repr hides payload/URL; Java/Python URL string representations are redacted.
Missing blocks vs explicit null	Missing defaults to empty; explicit null or null elements fail at validated entry points.
Python field replacement vs list mutation	Replacement is validated; append is supported and does not trigger Pydantic validation.

Behavioral contracts

1. Java structured-output parsing adds the parsed result without discarding the original message content or tool calls.
2. The Python FAIL debug message does not include input message bodies; source representations omit raw Base64/URLs.
3. Media string fields reject numeric/boolean coercion. Optional size_bytes accepts only integers in [0, 2^63 - 1] or null.
4. Java constructors/setters/map conversion reject null block containers/elements. Python construction and field replacement reject them; construction copies the input list and valid append remains supported.
5. Python prompt maps may omit blocks, but explicit null is rejected.
6. Base64 contents remain unchanged through serialization, with no encoding validation. Inferred sizes are non-negative and exact for valid standard Base64 with optional padding.

Failure behavior

Invalid media fields and block collections raise construction/deserialization errors rather than being coerced or dropped. Failed block replacement leaves existing blocks unchanged. Python validate_assignment applies to every ChatMessage field, not only blocks. Structured-output parse errors and chat retry policy are unchanged. This is not blanket log sanitization: arbitrary exception text and generic maps are outside these guarantees. Restoring Python-originated built-in Event types remains separately tracked in #1125.

Tests

Contract	Regression coverage
1: Preserve structured output content	Java structuredOutputPreservesOriginalBlocks; Python test_structured_output_preserves_original_blocks
2: Safe diagnostic output	Python chat FAIL/caplog regression and test_media_representations_hide_payloads_but_wire_preserves_them; Java base64PayloadPreservationAndSafeRepresentations
3: Strict media fields	Java mediaFieldsRejectScalarCoercionOnJsonAndMapPaths; Python media-string/source/size parameterized tests
4: Block collection boundaries	Java nullBlocksFailAtEveryEntryPoint; Python null construction/assignment and list-copy/append/dump tests
5: Prompt missing/null distinction	Java testOmittedBlocksDefaultToEmptyButExplicitNullIsRejected
6: Opaque Base64 and inferred size	Java payload-preservation/size test; Python test_base64_preserves_unvalidated_payload and test_base64_size_and_wire_preservation

Coverage focuses on wire compatibility, map conversion, accidental media disclosure, and content preservation. Python API/Plan/runtime-tests regression: 1046 passed, 13 skipped. Final Java targeted regression: 122 tests, 11 skipped, zero failures/errors, covering message JSON, structured output, prompt maps, resource adapter, Event Log, and state serde.

Not verified: live provider calls, full E2E jobs, skipped cross-language snapshot cases, non-empty tool-call preservation during structured-output parsing, exhaustive assignment validation for non-block fields, or generic-map Event Log sanitization. No provider converter or Event-type restoration is implemented here.

Verification commands and implementation details

• mvn test -pl runtime -am -Dtest=ChatMessageSerializationTest,ChatMessageTest,ChatModelActionTest,PythonPromptTest,JavaResourceAdapterTest,FileEventLoggerTest,ActionStateSerdeTest,CrossLanguageEventSnapshotTest -Dsurefire.failIfNoSpecifiedTests=false (JDK 17; rebuilds changed sources).
• PYTHONPATH=<venv-site-packages> python/.venv/bin/pytest -q python/flink_agents/api python/flink_agents/plan python/flink_agents/runtime/tests -m 'not integration'.
• Changed Python files pass Ruff; Java changes are Spotless-formatted; git diff --check passes.
• Java block lists remain immutable snapshots. Bridge construction initializes an empty list before applying validated map blocks. Python JSON and model_dump() continue emitting lists.

API

Wire shape is unchanged. Inputs relying on scalar coercion, invalid sizes, or Java null block lists are now rejected; Python field reassignment is now validated. Python list append remains available. Java structured output retains original text formatting instead of replacing it with cleaned JSON text. URL diagnostic strings become redacted. Normal JSON/state/bridge serialization retains the original media data and URL.

Documentation

• doc-needed
• doc-not-needed
• doc-included

Was this patch authored or co-authored using generative AI tooling?

• Yes
• No

Generated-by: Codex (model/version not exposed)

[Zhuoxi2000]

Structured output now matches Python; the message validation looks consistent. Could MediaFieldDeserializers be package-private? Jackson still instantiates its nested deserializers.

[wenjin272]

Thanks @Zhuoxi2000! Made MediaFieldDeserializers package-private, keeping its nested deserializers public. All 23 targeted message tests pass, including JSON and map deserialization coverage. Updated in 3e06a62.