Understand media (structured)

Label or extract structured information from images, audio, and video with a frontier multimodal model. You supply the instructions (prompt), the media (inputs), and a JSON schema; the response output conforms to that schema — closed label sets, scores, timestamps, transcribed fields, whatever you define. Synchronous. Billing follows the upstream model's token usage (input media and prompt, plus output including reasoning) at 1 credit per $0.10, rounded up per call, charged only on success. Per-key concurrency is limited during beta; requests beyond the limit are rejected, not queued.

Headers

X-API-Keystringrequired

Your VisionStory API key (sk-vs-...), kept server-side. Create or manage keys in API keys (Pro plan and up).

Request body

application/jsonrequired

inputsMediaRef[]required

1-8 media items (image / audio / video), each as asset_id, public url, or inline_data. Mixed kinds are fine. Not stored beyond the request.

asset_idstring | nullnullable

Asset ID from POST /api/v1/asset; use for materials reused across requests.

inline_dataInlineDataModel | nullnullable

Inline base64 media data for one-off use; not added to your asset library. Images: image/jpeg, image/jpg, image/png, image/webp, image/bmp, image/tiff, image/gif; audio: audio/wav, audio/x-wav, audio/wave, audio/mpeg, audio/mp3; video: video/mp4, video/quicktime, video/mov.

datastringrequired

The file's raw bytes encoded as a base64 string (no data: URI prefix).

mime_typestringrequired

MIME type of the inline data; the gateway uses it to tell image / audio / video apart. The accepted set depends on the endpoint — see the field carrying this object. Talking video accepts audio ['audio/avi', 'audio/mpeg', 'audio/mp3', 'audio/mp4', 'audio/m4a', 'audio/wav'] and images ['image/jpeg', 'image/jpg', 'image/png', 'image/webp', 'image/heic'].

urlstring | nullnullable

Publicly accessible media URL for one-off use; not added to your asset library.

promptstringrequired

What to extract or label. Describe the task and, if helpful, the meaning of each schema field; the model sees the media in inputs in the given order.

schemaobjectrequired

JSON Schema (draft 2020-12 subset) the output must conform to. Top level must be an object. Keep it flat and explicit — enums for closed label sets, description on each field.

Responses

200Successful Response

application/json

dataMediaUnderstandResponse | nullrequirednullable

The endpoint-specific payload. See this endpoint's response schema for its fields; null only when the operation returns no payload.

cost_creditintegerrequired

Credits charged for this call (usage priced at upstream rates, rounded up to a whole credit, minimum 1).

outputobjectrequired

The structured result, conforming to the request schema.

usageMediaUnderstandUsagerequired

input_tokensinteger

Tokens consumed by the prompt and media inputs.

Default: 0

output_tokensinteger

Tokens generated (including the model's reasoning).

Default: 0

messagestring

Human-readable status message; "success" on a successful call.

Default: success

server_timestring · date-timerequired

Server-side timestamp when the response was produced, in ISO 8601 format (UTC).

defaultError response. All failures share one envelope: an error object with a numeric code, a human-readable message, an optional details string, and an optional hint giving an actionable next step (useful for AI agents).

application/json

errorErrorDetailrequired

codeintegerrequired

Machine-readable error code. Mirrors the HTTP status for transport-level failures (e.g. 401, 404, 422, 500) and may carry a business-specific code otherwise.

detailsstring | nullnullable

Optional structured detail about the failure, e.g. a JSON string of per-field validation errors on a 422. Absent when there is nothing extra to report.

hintstring | nullnullable

Actionable next step for resolving the error, written for both humans and AI agents (e.g. how to fix the request, or where to obtain an API key). May be absent.

messagestringrequired

Human-readable explanation of what went wrong. Safe to log or surface to end users; not localized.