Structured Media Understanding API

Use POST /api/v1/media/understand to extract structured data from media, such as a subject label, scene description, or fields for your application. Supply a prompt, media inputs, and your output JSON Schema. This is a synchronous API, not a video generation task: there is no task ID to poll, public model selector, or free-text response mode.

Request parameters

Field	Required	Contract
`prompt`	Yes	String, 1–5000 characters
`inputs`	Yes	Array of 1–8 media references
`schema`	Yes	JSON Schema (supported 2020-12 subset) describing a top-level object

Each media reference supplies exactly one of url (publicly accessible), asset_id (an uploaded asset), or inline_data (mime_type and base64 data). Supported inline media includes JPEG, PNG, WebP, BMP, TIFF, GIF; WAV/MP3 audio; and MP4/MOV video. See the request schema for precise MIME types. Do not send local file paths as URLs.

Extract structured data with Python

After configuring authentication, replace the example URL with your own accessible media. The SDK sets a 180-second timeout for this operation.

Python

from visionstory import VisionStoryClient

client = VisionStoryClient.from_env()
result = client.understand_media(
    prompt="Identify the main subject in the image.",
    inputs=[{"url": "https://example.com/photo.jpg"}],
    schema={
        "type": "object",
        "properties": {"subject": {"type": "string"}},
        "required": ["subject"],
        "additionalProperties": False,
    },
)
print(result["output"])
print(result["usage"], result["cost_credit"])

CLI and MCP

The CLI accepts JSON for both --inputs and --schema:

Shell

visionstory understand-media \
  --prompt "Identify the main subject in the image." \
  --inputs '[{"url":"https://example.com/photo.jpg"}]' \
  --schema '{"type":"object","properties":{"subject":{"type":"string"}},"required":["subject"],"additionalProperties":false}'

Both local and remote MCP expose understand_media with prompt, inputs, and schema. Remote MCP accepts public URLs and asset IDs, not local paths or inline media. The local Agent Skill helper supports the same understand-media command as the CLI.

Response, billing, and failure handling

A successful REST response contains data.output (the structured object), data.usage.input_tokens, data.usage.output_tokens, and data.cost_credit. The SDK unwraps data automatically. This operation can take up to 180 seconds; allow sufficient time in the calling application or MCP client as well.

Only successful requests are charged. Cost is based on token usage, converted at 1 credit per USD 0.10 and rounded up, with a minimum of 1 credit. There is a per-key concurrency limit; excess requests are rejected rather than queued. Content moderation can reject a request with code 37100.

Do not automatically repeat a timed-out request: the server may already have succeeded, and another successful call can incur another charge. Inspect the error before retrying. Never log API keys or base64 media.

Speech to Text — use transcription for speech, SRT subtitles, and word timestamps.
Assets API — upload media once and reuse the returned asset_id.
For agents — connect an MCP client or install the Agent Skill.

Request parameters

Extract structured data with Python

CLI and MCP

Response, billing, and failure handling

Related guides