How AI answers phone calls

It's a pipeline, not magic

When an AI answers a phone call, there is no single "AI" doing the listening, thinking, and speaking. There's a pipeline of specialized components passing work to each other in real time: audio in, transcription, reasoning, speech out. Each stage is a hard engineering problem on its own; the product is making them feel like one continuous conversation.

This guide walks the real pipeline — the same architecture every serious voice AI uses — explains what each stage actually does, where the hard parts are, and how our own stack at esimgomobile is built. No invented latency figures, no accuracy claims: just the mechanics, honestly described.

What each stage does

1. PSTN / SIP ingress. The call arrives from the telephone network as a stream of audio packets (usually RTP). A media server terminates the call, decodes the audio, and hands frames to the pipeline — while simultaneously taking synthesized audio back and encoding it toward the caller. This stage is pure telecommunications plumbing: jitter buffers to smooth out network wobble, echo cancellation so the AI doesn't hear itself, and codec handling.

2. Voice activity detection (VAD). Before anything intelligent happens, the system must know when someone is speaking. VAD is a small, fast model that labels each slice of audio as speech or silence. It drives everything downstream: when to start transcribing, when the caller has finished, whether to keep listening or begin responding. Bad VAD means the AI talks over people or waits awkwardly — most "the AI feels off" complaints trace back here, not to the language model.

3. Speech-to-text (streaming ASR). Transcription runs continuously as the caller speaks, producing partial transcripts that get revised as more audio arrives ("I need a… I need a crown" converges on the final wording). The Whisper family of models set the modern bar for this: trained on vast multilingual audio, robust to accents and noise. Our stack uses whisper-large-v3-turbo — the speed-optimized variant — because in a live call, a transcript that arrives after the caller finished is a transcript that arrived too late.

4. Turn manager. Human conversation has choreography: we signal we're done speaking with intonation, pauses, and filler words, and we constantly negotiate who talks next. The turn manager replicates this — deciding when the caller has finished (endpointing), when to jump in, and when to hold back. It weighs pause length, intonation, sentence completeness, and whether the transcript forms a complete thought. Too eager and it interrupts; too cautious and every exchange has a dead second.

5. Language model. The conversation brain. It receives the transcript plus the conversation history, your business knowledge, and the tools it can call (calendar, CRM, transfer), then decides: what to say, and what to do. Our stack uses Qwen3 — a genuinely open-weight model — as the default and only conversation brain. Open matters here: the model's behavior can be inspected, the weights aren't rented from a single vendor's API, and the system doesn't inherit a closed provider's usage policies or pricing whims.

6. Text-to-speech (streaming TTS). The model's reply text is converted to audio as it's generated — the first chunk of speech starts playing while the rest of the sentence is still being written. This streaming is what makes the AI feel responsive instead of "thinking" for seconds between turns. Neural voices are now good enough that most callers focus on what's said, not how it sounds.

Anatomy of one back-and-forth

Zoom into a single exchange — caller asks, AI answers — and watch the stages overlap:

The caller starts speaking. Audio frames flow to the ASR, which emits partial transcripts — rough at first, converging as context grows. Once the turn manager judges the thought complete (a pause plus a complete sentence, roughly), the language model begins generating. It doesn't wait to finish: tokens stream to the TTS, which starts synthesizing immediately, and the first audio chunk plays to the caller while the rest of the sentence is still being formed.

Meanwhile the ASR keeps listening, because the caller might interject — which brings us to the hardest interaction in voice AI.

Barge-in: the moment that makes or breaks trust

In human conversation, we interrupt constantly — "yeah, Thursday works—" — and nobody finds it rude in context. An AI that can't be interrupted feels like a recording; an AI that handles interruptions gracefully feels alive.

The mechanics: while TTS audio plays to the caller, the VAD keeps watching the inbound audio. Speech during playback triggers a stop signal — the TTS halts mid-word (cleanly, not with a click), the partial transcript of the interruption goes to the language model, and a fresh reply begins. The subtlety is in the tuning: a cough shouldn't nuke the AI's sentence, but "no wait—" must. Good systems use a short confirmation window — tens of milliseconds of sustained speech — before committing to the stop.

There's a second, subtler case: the AI interrupting the caller. Humans do this with backchannels ("mm-hm," "right") that signal attention without taking the floor. Voice AI is learning the same trick — brief acknowledgments that don't trigger a full turn swap. It's one of the active frontiers; expect rapid improvement.

Phone audio is hostile. Plan for it.

Everything above was designed for the worst audio medium still in common use. Respecting that shapes every engineering choice:

Narrowband audio. A phone call carries roughly 8,000 samples per second — a fraction of studio audio, with frequencies above ~4 kHz simply gone. "S" and "f" blur; names get mangled. Transcription models must be trained or tuned for this, and critical details (names, numbers, addresses) should always be read back for confirmation.

Packet loss and jitter. Voice travels as packets across the internet before it ever reaches the AI. Packets arrive late, out of order, or not at all. Jitter buffers and packet-loss concealment hide most of it, but the pipeline must degrade gracefully when they can't — a brief robotic glitch beats a dropped call.

DTMF. Keypad tones still carry meaning — "press 1 to confirm" survives because it's unambiguous. Good systems accept both spoken and keypad input for confirmations.

Half-duplex callers. Many callers are on speakerphone in cars, kitchens, and construction sites. Echo cancellation and noise suppression aren't luxuries; they're the difference between a transcript and garbage.

Designed recoveries for every failure

Honest voice AI engineering spends as much time on failure as on the happy path. Every stage can fail, and each failure gets a designed recovery — not a crash, not a hallucination, a recovery.

The pattern across all of them: detect, acknowledge, recover, escalate. Detect that something went wrong (low transcription confidence, a silent caller, a question outside the knowledge base). Acknowledge it out loud — "Sorry, I didn't catch that" beats bluffing. Recover with a concrete next step — a repeat, a rephrase, a narrower question. And escalate to a human before the caller's patience runs out, with the full context attached.

One more honest note: no vendor — including us — can give you a single "accuracy" number that means anything. Accuracy depends on audio quality, accents, domain vocabulary, and what counts as an error. Ask instead for the recovery behaviors and for sample transcripts with mistakes in them. The mistakes tell you more than the successes.

How esimgomobile builds it: open models, hosted inference

Tele Agent — our AI receptionist, launching on esimgomobile — is built on two deliberate choices:

Open models, not rented brains. Qwen3 handles conversation; whisper-large-v3-turbo handles transcription. Both are open weights: inspectable, portable, and free of any single vendor's API terms. If a better open model appears next year, the stack can adopt it without renegotiating anything.

Hosted inference, not vendor lock-in. The models are served through hosted inference providers — specialized GPU infrastructure — rather than through a vertically integrated AI vendor. And to be explicit: there is no Groq anywhere in this stack, by decision. The models are free open weights; paying a premium middleman for the privilege of running them made no sense for a product whose costs must scale to millions of minutes.

Above the models sits the part that actually differentiates: your business knowledge, your calendar and tools, your escalation rules, and the failure recoveries described above — all configured per business, all under your control.

Before you ask.