Voice Agent — Architecture Reference

Last updated: 2026-06-29
Status: Architecture decided. Implementation deferred to Phase 16.

What Voice Agent Is

A voice channel for CE where contacts speak and the bot responds in audio. Same CE intelligence (CI memory, intent routing, model routing, tool registry) — different transport and modality.

Not a separate service. Voice is a new channel type added to the existing CE worker.

Session Model

Voice sessions are separate from text sessions.

• A voice call creates a new session with channel: "voice_web" or channel: "voice_phone"
• The contact's text chat sessions remain separate in the inbox
• CI bridges the gap: both voice and text sessions ingest to the same CI contact. Long-term memory, relationship stage, behavioral guidance — all carry over between channels automatically
• Human agent inbox shows voice calls and text chats as distinct items

Why Custom Pipeline (Not OpenAI Realtime API)

OpenAI Realtime API handles STT + LLM + TTS in one WebSocket. Fast and simple.

We don't use it because:

• Locked to OpenAI models — can't inject CI context, memory tiers, intent routing
• Can't inject L1 behavioral guidance from CI into the LLM call
• Can't do cost-optimized model routing (Groq for voice speed)
• The entire relationship intelligence vlozi has built becomes irrelevant

We build a custom chained pipeline — more work upfront, full control forever.

Voice Pipeline

User speaks
    ↓  WebSocket (16kHz PCM or Opus audio chunks)
CF Worker — WebSocket handler (/voice/ws/:channelId)
    ↓
Deepgram Nova-2  (streaming STT)
    ↓  partial transcripts → endpointing detects utterance end (~300ms pause)
    ↓  ~100ms to first partial transcript
CE Voice Orchestrator
    ├── DO.getSession()                        ~0.2ms (same DO hot path as text)
    ├── Promise.all:
    │     ├── fetchCIContext() if Tier 3+      ~30ms
    │     └── classifyIntent() via Haiku       ~20ms
    └── Groq Llama 3.1 70B (streaming LLM)    ~50ms to first token
    ↓  first LLM tokens arrive
Cartesia Sonic  (streaming TTS)               ~90ms to first audio chunk
    ↓  audio chunks back to client WebSocket
User hears first word

Total to first audio word: ~270ms — well within 500ms human comfort threshold.

Provider Decisions

STT — Deepgram Nova-2

• Best real-time streaming accuracy
• Built-in endpointing — detects when user stops speaking (configurable silence threshold, default ~300ms)
• Disfluency removal — strips "um", "uh", false starts before passing to LLM
• Cost: $0.0043/minute
• CF Workers: HTTP WebSocket to Deepgram's streaming endpoint

LLM — Groq (Llama 3.1 70B)

• Time to first token: ~50ms
• Compare: Haiku ~80ms, Sonnet ~300ms, Opus ~500ms
• Sonnet alone eats 60% of the voice latency budget — too slow
• Groq API is OpenAI-compatible — minimal integration work
• Voice channel config always routes to Groq, regardless of primaryModel

TTS — Cartesia Sonic

• Lowest latency TTS available: ~90ms to first audio chunk
• Streaming: audio chunks arrive before full sentence is complete
• Cost: $15/1M characters (~$0.018/minute of conversation)
• ElevenLabs Turbo v2 as backup: ~120ms, higher expressiveness, higher cost

Fallbacks

Cost per Minute of Voice

Typical 5-minute support call: ~$0.12 per call. Competitive vs traditional IVR.

Barge-In (Interruption Handling)

Without barge-in, the bot talks over the user — feels broken.

Flow:

1. Client runs VAD (Voice Activity Detection) — detects user voice while bot is speaking
2. Client sends { type: "barge_in" } over WebSocket
3. Server: AbortController.abort() cancels LLM stream
4. Server: sends { type: "stop_audio" } — client stops playing immediately
5. Deepgram continues transcribing the new utterance
6. New pipeline fires with the user's interruption
7. Previous incomplete response is discarded (not saved to session)

Client-side VAD options:

• Silero VAD (lightweight, browser, good accuracy)
• WebRTC VAD (browser-native, less accurate, zero dependency)

Voice-Specific System Prompt

Same assembleSystemPrompt() function. When channel = "voice_web" or "voice_phone", inject after persona:

You are speaking aloud in a real-time voice conversation.
- Keep every response to 1-2 short sentences maximum. Never more.
- Never use bullet points, headers, markdown, numbers, or any formatting.
- Be natural and conversational. Use contractions. Match the user's energy.
- If you need a moment, say "Let me check that for you" or "One moment."
- Never read URLs, order IDs, or long strings aloud. Say "I'll send that to you in the chat."
- When escalating to a human, say "Let me connect you with someone right now."

New Config Fields (BotConfig)

// Voice settings — added in Phase 16
voiceEnabled: boolean
voiceSttProvider: "deepgram" | "cloudflare_ai"
voiceTtsProvider: "cartesia" | "elevenlabs"
voiceLlmProvider: "groq" | "haiku"            // voice always needs fast model
voiceBargeInEnabled: boolean
voiceResponseStyle: "brief" | "conversational"
voiceSilenceThresholdMs: number               // endpointing sensitivity (default 300ms)
voiceMaxCallDurationMs?: number               // hard cap on call length

New Files (Phase 16A)

apps/chat-engine/src/
├── lib/
│   ├── voice-orchestrator.ts   STT→CE→TTS pipeline, barge-in, session management
│   ├── deepgram.ts             Streaming STT client
│   └── cartesia.ts             Streaming TTS client
└── routes/
    └── voice.route.ts          WebSocket upgrade handler (/voice/ws/:channelId)

New wrangler bindings:

DEEPGRAM_API_KEY = ""
CARTESIA_API_KEY = ""
GROQ_API_KEY = ""

New channel types in ce_channels schema: voice_web, voice_phone

What Does NOT Change

• Session storage — voice will use the Phase 6B SQLite DO once it exists; until then it uses the same Neon path as text (there is no DO message store today — see vision.md "The DO reality")
• D1 message archive — voice transcripts flush to D1 identically
• CI ingest — voice transcripts ingest to CI exactly like text messages
• Memory tiers — Tier 0-4 all work for voice
• Billing accumulator — canChatProceed() still runs before voice pipeline starts
• Handoff — [[HANDOFF]] still triggers escalation (warm transfer added in Phase 16B)

Latency Budget

Buffer: 230–300ms headroom before hitting 500ms threshold.

Phase Breakdown

Phase 16A — Web Voice

• Browser WebSocket endpoint at /voice/ws/:channelId
• Deepgram STT, Groq LLM, Cartesia TTS integration
• Barge-in (Silero VAD on client + server-side AbortController)
• Voice-adapted system prompt injection
• Separate session model, channel: "voice_web"

Phase 16B — Phone Voice

• Twilio Programmable Voice webhook → CF Worker
• PSTN audio: 8kHz G.711 (phone quality)
• Real phone number per tenant channel
• Warm call transfer to human agent via Twilio
• SMS fallback for URLs and long strings (instead of "I'll send it in chat")

Phase 16C — Voice + Advanced CI

• Crisis detection → voice-specific response + immediate warm transfer
• Proactive voice outreach (CE-initiated calls for Tier 4 contacts)
• Per-contact language detection → voice responds in their language
• Preferred call hours inference from CI interaction log

What Voice Is NOT

• Not a replacement for text chat — separate channel, separate use case
• Not real-time translation (language adaptation in Phase 16C)
• Not voicemail (real-time only)
• Not a phone system out of the box (Phase 16A = browser; phone = Phase 16B)
• Not multimodal (images/files in voice calls are out of scope)