What Is a Conversational Video API?

A conversational video API is an interface that lets your software join a live video call as a participant rather than as a recorder. An AI persona connects to the session, hears the person, sees their camera and shared screen, takes turns in real time, pursues the goals you configured, and returns a structured result when the call ends — transcript, scores, recording. It differs from a video generation API, which produces a clip to watch later, and from a meeting bot, which sits silently and takes notes.

That paragraph is the whole definition. The rest of this post is about which parts of the stack decide whether one actually works, because the category's marketing has trained everyone to evaluate the wrong one.

The category was defined by the wrong feature

Conversational video got its public shape from avatar rendering. The demos were faces — photoreal, well-lit, uncannily smooth — and so the evaluation criteria became face quality.

Two years of watching people talk to these things has convinced me that's backwards. Participants adjust to an imperfect face within about thirty seconds. What they never adjust to is being interrupted mid-sentence, or a persona that forgets who they are between sessions, or a thirty-minute conversation that produces nothing you can act on.

The face is the cheapest part of the stack. The conversation is the expensive part. If you're evaluating vendors, weight accordingly.

The seven parts

Every vendor in this category has all seven. The differences are entirely in which ones they've invested in, and the two that determine whether users come back are the two at the bottom.

Turn-taking is the whole product

Getting a model to say a sentence is solved. Getting it to know when not to is not.

Human conversation runs on gaps measured in a couple hundred milliseconds. Too eager and the agent steps on people; too slow and the silence reads as broken software and the person starts talking again, which produces a collision anyway. And it isn't a fixed threshold — a pause mid-thought is different from a pause at the end of an answer, and the difference is in prosody, not duration.

Everyone who has built in this space has watched a user go quiet, get interrupted, and visibly disengage for the rest of the call. That's the failure mode that decides adoption, and it never shows up in a feature comparison table.

It's the reason we ended up building our own realtime speech-to-speech model, ORI-Realtime 1.5, currently in research preview, rather than assembling one from parts. When turn-taking is the product, you want the conversation loop to be yours.

Vision is what makes it video

If the AI can't see, you've built a phone call with a webcam attached. The distinction is worth being concrete about.

Through the camera, the persona reads facial expressions to "detect engagement, confusion, confidence, and emotional state," body language including "posture, gestures, and non-verbal communication," and the participant's environment and any physical objects they hold up. Through a shared screen it can read documents — "text, spreadsheets, and presentations" — recognise software interfaces and guide someone through a workflow, read code on screen and give feedback on it, and interpret charts, diagrams, and design mockups.

That last cluster is what makes technical interviews and product walkthroughs work at all. A candidate sharing an editor isn't an audio problem, and no transcript of what they said captures what they wrote.

One caveat that matters for your product design, quoted directly from our docs because vendors tend to skip past it: "Participants always control whether to turn on their camera or share their screen. Enabling these settings makes the option available — it doesn't force it." Build for the case where they decline. If your scoring silently depends on vision, it will produce nonsense the first time someone joins camera-off.

Worth knowing too: once vision is on, you don't have to prompt for it. The persona incorporates what it sees automatically rather than waiting to be told to look.

The result is what you're actually buying

Here's the part I'd push hardest on during an evaluation, because it's where the category is thinnest.

A conversation that ends in a transcript has moved your work, not finished it. Somebody still has to read it, judge it, and write the outcome into whichever system cares. At volume that's the entire cost.

A conversational video API worth integrating ends the call with structured output. In our case you define goals on the scenario, each with three parts: a name — "what you're measuring (e.g., 'Communication Clarity')" — a description of what good performance looks like, and scoring instructions telling the AI how to assign a score. Each goal is scored on a scale, typically 1 to 10, judged from the transcript, your scoring instructions, and observable behaviour during the session. What comes back is individual goal scores with written feedback, an overall average, and transcript highlights linked to the specific goal each one supports.

The practical advice from watching people write these: scoring instructions that describe observable behaviour work; abstract qualities don't. "Names a specific metric when asked how they'd measure success" produces consistent scores. "Demonstrates strategic thinking" produces noise.

That output is the difference between an interesting demo and something you can put in front of a hiring team.

What separates an API from a demo

Between the conversation and the result sits a set of unglamorous things that decide whether you can ship this:

Session lifecycle — create a meeting, get a join link, know its status, know when it ended.

Mid-call tool calls — the persona calls your code during the conversation, so it can look up an order, check a calendar, or write to your system while the person is still on the call. Without this the agent can only talk about your product; it can't operate it.

Participant memory — the persona recognises someone it has spoken to before. Second conversations that start from zero feel worse than no AI at all.

Consent and recording — a pre-join consent step, recording handled by the platform. Not optional, and not something you want to be building yourself.

Events — signed webhooks for meeting ended, transcript ready, analysis complete, recording ready. This is what turns a meeting into a row in your database without polling.

Multi-agent handoff — more than one agent in a session with handoffs mid-call, for flows where a specialist should take over.

If a vendor demos beautifully and can't answer questions about those six, you're looking at a demo.

The three build shapes

Almost every integration lands in one of these:

Hosted join link. You create a meeting, you get a URL, the participant joins from it. Nothing to build on the frontend. Best for interviews, screening, and anything sent by email.

Embedded widget. Mint a short-lived signed JWT, drop in the iframe snippet, and the conversation happens inside your product or LMS. Best when you want it at the moment of intent rather than in a separate tab.

Agent dispatched to a meeting. The persona takes the call on your behalf, by link or on a schedule. Best for the delegation use case — a call that needs to happen when you can't be there.

Same API underneath. The choice is a product decision about where the conversation belongs.

Frequently asked questions

What's the difference between a conversational video API and a video generation API? A generation API renders a clip you watch later — the output is a file. A conversational video API joins a live two-way session as a participant and the output is a conversation plus a structured result. Different products that share a word.

Does it need WebRTC? Underneath, yes — real-time bidirectional media is what WebRTC is for. Whether you touch it depends on your build shape. With a hosted join link or an embed you never see it; if you're building your own frontend media surface, you will.

Can the AI actually see a shared screen? Yes. It reads documents, spreadsheets, presentations, application interfaces, code, and diagrams from a screen share, and expressions and body language from the camera. Both are participant-controlled — they choose whether to enable them.

Is a conversational video API the same as an AI meeting API? Close enough to overlap, with a difference of emphasis. "Conversational video" describes the primitive — a real-time two-way video conversation with an AI. "AI meeting API" describes the whole lifecycle around it: scenarios, personas, goals, sessions, transcripts, analyses, recordings.

What comes back after the call? Transcript, an analysis scoring the participant against your goals with written feedback and highlights, and a recording. Delivered by webhook as each one becomes available rather than all at once.

Deeper on the category in what an AI meeting API actually is, or start building from the API quickstart — two calls to a first meeting.