Chat Engine — Durable Object Hot Path (Neon scale-to-zero)

Last updated: 2026-07-01 Status: Built, flag-gated (CE_HOTPATH_DO), NOT enabled in prod. Validate one tenant before rollout.

Problem

chat-engine (ce_*), contact-intelligence (ci_*), and brain (ai_*) all share one Neon compute (ep-misty-lake / neondb). Neon suspends only after ~5 min with zero queries, but every chat message fans out into a burst of queries on that one compute — CE (session/config/history/persist) + two CI /ingest calls + an optional knowledge search. So one message every few minutes keeps the compute permanently warm, which is billable. There is no polling bug; it's the aggregate of per-message traffic.

Solution

Run the per-message hot path off a per-session Durable Object (SessionStore) whose own SQLite holds the active conversation, and flush to Neon in one batch when the conversation goes idle. Active conversation = zero Neon queries; quiet = one write then the compute suspends.

Components

• SessionStore DO (src/session-store.ts) — a new class (the live SessionScheduler SLA DO is untouched). Per-session SQLite (msg + kv):
  • POST /load → history + summary + messageCount + status, and a TTL-cached bot config + custom tools (loads from Neon only on miss/stale, ~every 10 min).
  • POST /append → writes the turn to SQLite; idle-flush alarm (2 min) for AI sessions, immediate flush once escalated, forced flush at 20 buffered messages.
  • flush() — serialized (mutex); batch-upserts the session + unflushed messages to Neon, batches the CI ingest in the same pass, then marks rows flushed. Ordered by rowid (monotonic). The upsert never overwrites status/closedAt (owned by the escalate/resolve routes).
• Client (src/lib/session-store-client.ts) — loadWorkingSet / appendTurn; both degrade to null/false so a DO hiccup falls back to Neon.
• Orchestrator (src/lib/orchestrator.ts) — prepareTurn acquires the working set via acquireViaDO (hot path) or acquireViaNeon (default); persistTurn writes via the DO or Neon. CI ingest is skipped per-message when the hot path is on (the DO batches it).

The flag

CE_HOTPATH_DO="1" (env var) routes the hot path through the DO. Unset/anything else → the Neon path runs exactly as before. The DO is bound (SESSION_STORE, migration v2) but inert until the flag is set.

What still wakes Neon (accepted)

• Tier-3 (CI relationship) bots read CI context inline per message — can't batch a read the response needs. ~1 of 6 prod tenants.
• Knowledge-enabled bots read ai_knowledge_chunks inline. Semantic search isn't cacheable.
• The DO's config/tools cache misses (~every 10 min per active session) and the idle-flush burst.

So the 5 Tier-2 bots get fully off the per-message Neon path; the compute suspends whenever their conversations are idle.

Tradeoff (decided)

Idle-flush means an active AI-handled conversation isn't in Neon until ~2 min idle, so the inbox lags for in-progress AI chats. The moment a session escalates, it live-flushes and stays live — so human-handled chats are always current. (Chosen over always-flush / short-interval.)

Review fixes already applied

Flush serialization mutex; status/closedAt not clobbered by flush; monotonic rowid ordering (no same-ms trim bug); escalated derived from status (not sticky); forced flush bounds buffer growth.

Rollout plan

1. Deploy with the flag unset (no behavior change; DO bound but inert).
2. Set CE_HOTPATH_DO=1 for a staging/one-tenant worker; run real chats.
3. Verify in Neon: the compute suspends between conversations; messages/sessions land on idle-flush; escalation flushes live; the inbox shows escalated chats immediately.
4. Confirm no regressions (history correct, summaries roll, tools work, billing settles).
5. Widen to all traffic; monitor Neon compute-hours drop.
6. Rollback = unset the flag (instant revert to Neon path).

Deferred

• DO storage cleanup after long idle (evict cold sessions).
• Separate CI-ingest retry tracking (currently best-effort in the flush pass, matching the old per-message behavior).