feat(multimodal): re-implement audio/video support on main

Re-implements PR #2908's generalized multimodal support on the
post-FSM main branch:

- Audio input: detects WAV/MP3/OGG/FLAC via magic bytes, sends
  input_audio blocks to compatible providers, falls back to
  [audio: path] placeholder when unsupported.
- Video input: sends video_url data-URI blocks to compatible
  providers, falls back to [video: path] placeholder.
- InputLimitsConfig: count limits (images/audios/videos) and byte
  limits per media type.
- AgentDefaults: pattern-matched vision_models, audio_models,
  video_models with supports_*() helpers.
- Provider retry: strips all media types (image_url, input_audio,
  video_url) on non-transient errors and retries once.
- Feishu: extracts media tags from post messages.
- Anthropic & OpenAI Responses converters handle audio/video.

Tests: 17 new multimodal tests + existing suite passes.
This commit is contained in:
chengyongru
2026-05-20 11:47:01 +08:00
parent 1391aa3d57
commit 0e754d2591
11 changed files with 625 additions and 71 deletions
+2 -2
View File
@@ -242,7 +242,7 @@ async def test_image_fallback_returns_error_on_second_failure() -> None:
@pytest.mark.asyncio
async def test_image_fallback_without_meta_uses_default_placeholder() -> None:
"""When _meta is absent, fallback placeholder is '[image omitted]'."""
"""When _meta is absent, fallback placeholder is '[image]'."""
provider = ScriptedProvider([
LLMResponse(content="error", finish_reason="error"),
LLMResponse(content="ok"),
@@ -256,7 +256,7 @@ async def test_image_fallback_without_meta_uses_default_placeholder() -> None:
for msg in msgs_on_retry:
content = msg.get("content")
if isinstance(content, list):
assert any("[image omitted]" in (b.get("text") or "") for b in content)
assert any("[image]" in (b.get("text") or "") for b in content)
@pytest.mark.asyncio