TL;DR: Achieving sub-400ms voice AI barge-in latency over SIP and Asterisk AudioSocket requires streaming raw 20ms signed linear PCM frames, running edge Voice Activity Detection (VAD) directly on the inbound media socket, immediately flushing queued outbound TTS buffers upon human speech onset, and syncing exact heard-word offsets into Memoria (100% LongMemEval).
Why Is Voice AI Barge-In Latency the Defining Metric for Natural Telephony?
In human telephone conversations, acoustic turn-taking happens within 200ms to 350ms, and callers routinely interrupt (“barge in”) when an agent reads an address they already know or suggests a calendar time that conflicts with their schedule. When voice AI barge-in latency exceeds 800ms, the AI continues talking over the human caller, creating awkward cross-talk collisions that immediately expose the system as a rigid bot and cause callers to hang up.
Most cloud voice wrappers suffer from 1,200ms to 2,200ms turn-taking delays because their telephony media path traverses multiple high-level abstractions: SIP RTP packets are transcoded into WebRTC, proxied over regional HTTP/WebSocket hops, buffered for cloud Speech-to-Text (STT) endpointing, passed to an LLM, synthesized via chunked Text-to-Speech (TTS), and queued inside a PBX jitter buffer that cannot be cleared synchronously. By eliminating proxy hops and anchoring directly to Asterisk AudioSocket and low-level SIP RTP streams, Kolivri keeps end-to-end barge-in arrest under 100ms and full conversational turn-taking below 400ms across all 8 supported languages (Hebrew, English, Arabic, Russian, French, Spanish, German, and Portuguese).
What Happens Inside the Media Pipeline During a Caller Interruption?
When a caller interrupts a speaking AI agent mid-sentence, four operations must execute deterministically inside a single 20ms frame window:
- Acoustic Echo & Duplex Separation: Because PSTN and cellular handsets occasionally leak acoustic echo back into the uplink channel, the inbound 16kHz PCM stream is evaluated against the active outbound reference frame so the agent never triggers false self-interruption.
- Sub-30ms Speech-Onset VAD: Rather than waiting 600ms for a full STT word transcript, an energy-and-spectral Voice Activity Detector evaluates 20ms
slin16frames and flags confirmed human speech onset within two consecutive frames (40ms). - Immediate Outbound AudioSocket Buffer Flush: The media controller issues an atomic clear signal to the Asterisk AudioSocket / RTP ring buffer, dropping all unplayed TTS frames immediately so the line goes silent in under 80ms.
- Transcript Truncation & Memoria State Alignment: Using the exact byte count of PCM audio actually transmitted before the interrupt, the orchestrator truncates the assistant’s turn at the exact word the caller heard and commits that verified state to Memoria. You can test how interruption-safe memory works in our live 2-Call Voice AI Amnesia Test.
How Does Asterisk AudioSocket Compare to WebRTC and REST Voice Pipelines?
Engineering teams evaluating telephony transport layers for autonomous voice agents can compare the architectural latency breakdown in the table below:
| Pipeline Stage / Metric | Kolivri Asterisk AudioSocket (slin16 TCP) |
Managed WebRTC Wrapper (Vapi / Retell Style) | Legacy HTTP Chunked TTS over TwiML |
|---|---|---|---|
| Inbound Media Framing | Raw 20ms PCM frames (320 bytes @ 8kHz / 640 bytes @ 16kHz) | Opus-to-PCM transcoding across browser/edge relays (60–120ms) | Buffered media streams over external TLS proxies (150–300ms) |
| Barge-In Stop Latency (TTS Arrest) | 40ms–85ms (atomic ring-buffer drop on socket) | 350ms–650ms (cross-region control message + jitter drain) | 1,100ms–2,400ms (must finish current audio file or chunk) |
| End-to-End Turn-Taking Latency | Sub-400ms (streaming STT partial + speculative prefill) | 650ms–1,100ms depending on external LLM/TTS vendor hops | 1,800ms–3,200ms sequential request/response waterfall |
| Interrupted Context Accuracy | Exact byte-offset word tracking synced to Memoria (100% LongMemEval) | Full unplayed LLM paragraph often logged as “spoken” in history | No visibility into partial playback position at hangup/interrupt |
| All-Inclusive Production Cost | Starter $69 (₪247), Business $139 (₪479), Pro $249 (₪880) | $0.15–$0.31/min stacked across 4 separate vendor invoices | High per-minute telephony + STT + TTS character overages |
Why Byte-Accurate Barge-In State Matters for Long-Term Agent Memory
A subtle failure mode in naive voice agents occurs when a caller interrupts the AI before it finishes stating a price, policy, or appointment slot. If the LLM generated a 40-word response (“Your technician is booked for Tuesday at 3:00 PM, and the diagnostic fee is $85”) and the caller interrupted after word six (“Wait, make it Wednesday”), a standard voice wrapper records the entire 40-word string into conversation history. On the next turn—or the next call—the bot falsely believes the caller was informed about the $85 fee.
Because Kolivri measures the exact number of PCM samples drained through the Asterisk AudioSocket prior to the barge-in event, it reconstructs the exact spoken prefix before indexing the turn into Memoria, which achieves 100% accuracy on LongMemEval in English and 98% on Hebrew LongMemEval. Developers orchestrating outbound calls from Claude Code or Cursor via the Kolivri Voice MCP server (@kolivri/voice-mcp) receive deterministic transcripts that reflect what actually transpired on the wire.
Deploy Sub-400ms Voice AI on Your Phone Line in 30 Seconds
You do not need to write C dialplan modules or tune Asterisk audiosocket.conf buffers manually to get sub-400ms turn-taking in production. Connect your existing number via 30-second conditional call forwarding (*61* to 054-787-5670) or SIP trunking at https://kolivri.com/start/, manage live transcripts in https://app.kolivri.co.il, and choose a predictable plan: Starter at $69/month (₪247, 100 minutes), Business at $139/month (₪479, 220 minutes), or Pro at $249/month (₪880, 450 minutes).
Frequently Asked Questions
What causes high voice AI barge-in latency on traditional SIP trunks?
High voice AI barge-in latency (often 1,200ms to 2,200ms) is caused by buffering RTP packets inside HTTP chunked transfers, waiting for cloud STT endpointing timeouts before halting TTS playback, and failing to issue an immediate outbound audio buffer flush over Asterisk AudioSocket or ARI when inbound energy crosses the VAD speech threshold.
How does Asterisk AudioSocket improve turn-taking speed compared to WebRTC bridges?
Asterisk AudioSocket streams raw signed linear PCM (slin16) frames over a persistent TCP socket with 20ms framing and zero transcoding hops. When the voice engine detects human speech onset, it drops queued outbound PCM frames in under 20ms, keeping total voice AI barge-in latency below 400ms.
What happens to interrupted sentences in agent memory when a caller barges in?
Kolivri tracks exact byte-offset playback timestamps so only words actually heard by the caller are committed to the conversation transcript and indexed into Memoria (100% LongMemEval accuracy). Unspoken TTS tokens are discarded, preventing the AI from assuming the caller heard information that was cut off.
How much does Kolivri cost with sub-400ms SIP turn-taking and Memoria included?
Kolivri provides all-inclusive plans across 8 languages: Starter at $69/month (₪247, 100 minutes), Business at $139/month (₪479, 220 minutes), and Pro at $249/month (₪880, 450 minutes). Every tier includes sub-400ms barge-in, Memoria long-term memory, and instant setup at https://app.kolivri.co.il.
Give your AI agents & IDEs deterministic long-term memory with Memoria
Connect Claude Code, Cursor, Windsurf, VS Code, Zed, OpenCode, and custom agents via 30 MCP tools and 8 lifecycle hooks. Start free with 1,000 API calls & memories.