Chat Engine — Product Vision

Last updated: 2026-06-29
Status: Architecture finalized. Phase 6A (intelligence layer, no storage change) ready; Phase 6B (DO/D1 storage) deferred. Verified against live source 2026-06-30.

What is Chat Engine?

Chat Engine is vlozi's customer conversation intelligence platform — the layer between a business's contacts and their AI bot, turning raw messages into meaningful, configurable, and measurable conversations.

Vision Statement

Every business should be able to deploy a conversational AI that matches their brand, adapts to their customers, and gets smarter over time — without writing a single line of code.

Design Principles

1. Configuration over code — every behavior is a setting, not a deployment
2. Progressive intelligence — start simple, unlock depth as the business grows
3. Cost-aware by design — LLM token cost is the dominant cost; every architectural decision optimizes for it
4. Resilient by default — the contact always gets a response, even when dependencies fail
5. Privacy as default — data is tenant-scoped; deep memory requires explicit opt-in
6. Voice-ready — text architecture is designed so voice is an additive layer, not a rewrite

Memory Tiers

Tier drives default settings. Every dimension can be individually overridden.

Storage Stack

CE today runs entirely on Neon (ce_bot_config, ce_channels, ce_sessions, ce_messages). That stays the default. The DO/D1 evolution below is Phase 6B — deferred, and justified by Neon cost/load at scale + voice-readiness, NOT by text latency (text response time is ~90% LLM; shaving DB ms is invisible to the user).

Phase 6B target (deferred)

SQLite DO  → active session working set    (new DO class — see note below)
D1         → message archive               ~5-10ms, ULID composite key
Neon       → sessions, config, channels    stays — complex queries / inbox
Analytics Engine → intent events, metrics  write ~1ms

The DO reality (important correction)

The current SessionScheduler DO (scheduler.ts) uses the KV-style ctx.storage API, not SQLite tables, and is instantiated only on escalation (it self-destructs after the SLA alarm fires). It cannot be "extended" into a per-message store:

• A DO's storage backend (KV vs SQLite) is fixed at class creation via wrangler new_sqlite_classes migration — you generally cannot add SQLite to an already-deployed KV-backed DO. Phase 6B needs a new SQLite-backed DO class.
• Making it the message hot path routes every message through a DO stub — a request-flow re-architecture, not an extension.
• Latency note: DO storage is sub-ms inside the DO, but the Worker→DO fetch hop is ~1–5ms in-colo. That beats Neon's ~15–30ms, but it is not "0.2ms" — and the LLM (~1.2s) dominates regardless. This is a cost/scale play, not a speed play.

D1 message archive (Phase 6B)

CREATE TABLE ce_messages (
  id         TEXT,     -- ULID: lexicographic sort = chronological order
  session_id TEXT,
  tenant_id  TEXT,
  role       TEXT,
  content    TEXT,
  intent     TEXT,
  model      TEXT,
  latency_ms INTEGER,
  PRIMARY KEY (session_id, id)
);
CREATE INDEX idx_tenant ON ce_messages (tenant_id, id DESC);

ULID as message ID (current code uses crypto.randomUUID() — ULID is a change): first 10 chars encode the millisecond timestamp, so lexicographic sort = chronological. ORDER BY id DESC LIMIT N on the composite key is a covering-index hit, O(1) regardless of total rows.

Neon (stays — the default today and after 6A)

ce_bot_config, ce_channels, ce_sessions, ce_messages — complex filters, joins, paginated inbox. Until 6B is justified, all message reads/writes stay on Neon, exactly as the orchestrator does today.

Cloudflare Analytics Engine (Phase 6B)

One event per message:

{ tenant_id, session_id, intent, model, latency_ms, resolved, escalated, crisis_level }

Dashboard aggregates (intent distribution, resolution rate, p50/p95) read from here — zero Neon cost for analytics.

Verified Against Code (2026-06-30)

A pass over the live CE source confirmed these gaps the roadmap targets:

session_count note: the orchestrator passes ce_sessions.messageCount to CI as session_count (orchestrator.ts:201). It increments per message, so it's really "messages so far this session," used as a rough cold-start proxy. Flagged for CI alignment, not a blocker.

Pipeline (Phase 6A)

Message arrives
    ↓
load session (summary + history + config)     ← Neon today; SQLite DO in 6B
    ↓
Promise.all([
  fetchCIContext(),   ← Tier 3+
  searchKnowledge(),  ← if enabled
  classifyIntent(),   ← always, Haiku
])
    ↓
model = routeModel(intent, crisis_level, memoryTier)
    ↓
LLM(system[cache_control], history, message, tools)
    ↓
Return response to contact
    ↓
waitUntil (non-blocking):
  ├── persist messages (Neon today; D1 batch in 6B)
  ├── update session (messageCount, lastMessageAt, summary)
  ├── populate ce_messages.intent
  └── AnalyticsEngine.writeEvent  (6B)

The Promise.all is a genuine, cheap win — but its value is keeping preprocessing cost flat as we add steps (intent, moderation), not cutting user-perceived latency. For text the LLM (~1.2s) dwarfs the ~30–40ms of preprocessing either way.

Intent → Model Routing

Every message is classified before the LLM sees it. Classification result drives model selection.

Intent labels: greeting | short_question | chitchat | complaint | product_question | order_status | purchase_intent | escalation_request | feedback | off_topic

Routing rules:

greeting | short_question | chitchat     → haiku   (10× cheaper)
complaint | purchase_intent | complex    → sonnet
crisis_level = distress | crisis        → sonnet   (always, no exception)
memoryTier = 0 | 1                      → haiku    (always, no deep reasoning needed)

70% of customer messages are simple. 70% Haiku + 30% Sonnet routing = ~90% LLM cost reduction vs all-Sonnet.

Prompt Caching

System prompt is identical across all messages for a tenant (persona + rules + knowledge).
Marking it with cache_control: { type: "ephemeral" } → Anthropic charges ~10% of normal rate on cache hits.

At 10M messages/month with a 500-token system prompt:

• Without caching: ~$15,000/mo in system prompt tokens
• With caching: ~$1,500/mo

Single most impactful cost change. But the change lives in @repo/brain, not CE. Today CE passes system as a plain string (llm.ts:67). cache_control must be attached to a structured system block inside the @repo/brain Anthropic provider and threaded through the shared LLMProvider interface. That package is used by other services — so this touches shared code and needs regression checks beyond CE.

Configuration System

interface BotConfig {
  // Memory
  memoryTier: 0 | 1 | 2 | 3 | 4
  historyWindow: 3 | 6 | 10 | 20
  summaryEnabled: boolean
  summaryDepth: "brief" | "detailed"
  ciEnabled: boolean                         // requires tier 3+
 
  // Model
  primaryModel: "haiku" | "sonnet" | "opus"
  fallbackModel: "haiku" | "sonnet"
  perChannelModel?: Record<string, string>   // e.g. { whatsapp: "haiku", widget: "sonnet" }
 
  // Context
  knowledgeEnabled: boolean
  knowledgeSources: string[]
  crisisDetectionEnabled: boolean
  moodAwarenessEnabled: boolean
  languageMode: "auto" | string
 
  // Behavior
  responseLength: "brief" | "balanced" | "detailed"
  tone: "formal" | "casual" | "friendly" | "professional"
  handoffTrigger: string
  maxDailyMessagesPerContact?: number
  contentModerationEnabled: boolean
  fallbackMessage: string
 
  // Channels
  enabledChannels: ChannelType[]
  perChannelResponseFormat?: Record<string, string>
  perChannelMaxLength?: Record<string, number>
 
  // Actions
  builtinTools: BuiltinTool[]
  customTools: CustomTool[]                  // webhook-configured, no code change needed
 
  // Proactive (tier 4 only)
  proactiveEnabled: boolean
  quietHoursStart?: number
  quietHoursEnd?: number
  maxOutreachPerDay?: number
  reEngagementThresholdDays?: number
}

Config is read once from Neon on first message, then cached in DO for the session lifetime.

Cost at Scale

Without optimization: ~$35,000/mo (all Sonnet, no caching).
Savings: 90% — almost entirely from LLM routing and prompt caching, not from DB architecture.

The DB architecture (DO + D1) does not drive cost savings — those come from LLM routing + caching, which need zero storage change. DO/D1 is about Neon load at scale and voice-readiness. That's why it's split into 6B below.

Key Capabilities (Phased)

Phase 6A — Intelligence Layer (Next, no storage change)

• Parallel preprocessing (Promise.all over CI + knowledge + intent)
• Intent classification on every message → populate ce_messages.intent
• Intent → model routing (Haiku vs Sonnet)
• Prompt caching in @repo/brain Anthropic provider (the real cost lever)
• Write ce_audit_log on config / channel / session changes
• COMMS alert on SLA breach (scheduler.ts:78)
• Consolidate processChat / streamChat duplication
• Circuit breaker on CI calls

Phase 6B — Storage Evolution (deferred — gate on Neon cost or voice)

• New SQLite-backed DO class (not the KV SessionScheduler) for the session working set
• D1 ce_messages with ULID composite key
• Analytics Engine events
• Justification: Neon cost/load + voice-readiness — not text latency

Phase 7 — Action Layer

• Tool registry (built-in + custom webhook tools)
• recall_memory, search_knowledge, collect_lead, escalate_to_human, submit_form
• Tenant-configured HTTP webhook tools via dashboard

Phase 8 — Proactive Sessions

• Outgoing session model (initiatedBy: "inbound" | "proactive")
• CF Queue consumer for outreach delivery
• CI B3 bug fix (outreach triggers now actually delivered)

Phase 9 — Human Collaboration

• copilot mode — AI drafts, human approves
• supervised mode — AI responds, human can inject
• Draft holding (role: "draft", not delivered until approved)

Phase 10 — Conversation Flows

• Flow graph data model (nodes + edges)
• Node types: message, question, tool_call, branch, handoff, end
• Session variables for slot filling
• Flow execution engine wrapping orchestrator

Phase 11 — Multimodal Input

• WhatsApp / Telegram: images → vision, audio → transcription, documents → text extraction
• All transformed to text before LLM pipeline

Phase 12 — Rich Response Formats

• WhatsApp interactive buttons and list messages
• Widget: markdown, quick replies, cards
• Per-channel format configuration

Phase 13 — Webhook Event System

• session.started, message.received, message.sent, session.escalated, session.resolved, lead.captured, intent.detected
• Tenant subscribes per event in dashboard

Phase 14 — Cross-Channel Continuity

• Same contact on WhatsApp + widget = one session, shared CI memory
• Uses CI ci_contact_channels merge model

Phase 15 — A/B Testing + CSAT

• Persona variant testing with outcome tracking
• Optional CSAT rating on session close

Phase 16 — Voice Agent

• WebSocket handler (bidirectional, replaces SSE for voice)
• STT integration (Deepgram Nova-2)
• TTS integration (Cartesia Sonic or ElevenLabs Turbo)
• Barge-in / interruption handling
• Voice-optimized model routing (Groq Llama for lowest TTFT)
• Separate architecture doc: docs/chat-engine/voice-agent.md

What Is Intentionally Preserved

• Billing accumulator pattern — canChatProceed() + accumulateAndMaybeSettleChatCost() works well
• Durable Object SLA scheduler — extend with COMMS alert on breach, don't replace
• Session-per-channel-key model (wa-{phone}) — stable, simple, no change
• Hono router structure — add routes, don't restructure
• Neon for sessions + config — D1 migration deferred; risk not worth it now

What Is NOT in CE's Scope

• Contact Intelligence (memory extraction, relationship tracking) — separate service
• Content scheduling, blog, forms — separate services; CE calls them as tools
• Authentication — handled by gateway before CE ever sees a request
• Media storage — CE receives media URLs from channels, passes to media-service