Full-repo correctness + security audit of the credit/billing system, run after the addon-lifecycle hardening, seat-pricing enforcement, and value-based infra-pricing pass. Three independent read-only reviews (core ledger primitives, the manager billing lifecycle, and every service charge site) plus direct read-only queries against the live prod database. Every Critical and exploitable High below was hand-verified against the actual code (or, for the integrity-audit finding, against live prod data) before being recorded — no finding here is relayed without independent confirmation.
Status column: FIXED = shipped and tested in this pass; OPEN = confirmed,
not yet fixed; SUSPECTED = plausible from reading, needs live confirmation.
🔴 CRITICAL
C1. /comms/v1/send trusted tenantId from the request body — impersonation + cross-tenant spend — FIXED
File: apps/communication/src/index.ts (runBilledSend, POST /v1/send)
runBilledSend did const tenantId = body.tenantId ?? "system". The /v1/send
handler never overrode it with the verified x-tenant-id header. Any workspace
member holding comms:send could:
- POST
{tenantId: "system"}→isSystemTenantbypasses the rate limiter, credit gate, and accumulator, and the mail sends from the official platform identity (rootvlozi.app/ default platform sender). - POST
{tenantId: "<victim>"}→ charges the victim's accumulator and sends from the victim's verified alias/domain (ownership checks evaluated against the spoofed tenant, not the real caller).
Why it wasn't closed elsewhere: the gateway strips the client's
x-tenant-id header and re-stamps it from the verified JWT
(apps/gateway/src/utils/proxy.utils.ts), but that only protects the header —
nothing touched the request body, and sendSchema.tenantId was
z.string().optional().
Fix: POST /v1/send now checks whether x-user-permissions is present
(the marker the gateway always stamps alongside x-tenant-id for a real user
JWT — buildDownstreamHeaders sets both from the same ctx.auth block, never
one without the other). When present, body.tenantId is unconditionally
overridden with the header value; a user-permissions call with no tenant
header is rejected (400) rather than falling back to the "system" default.
When x-user-permissions is absent — the internal/system-caller case (e.g.
manager's CommunicationClient sending welcome/invite mail via the service
binding with the gateway key but no user JWT) — body.tenantId is trusted
exactly as before, preserving the legitimate "system" default for
platform-level mail.
Tests: apps/communication/test/send-tenant-spoof.test.ts (new, 5 tests) —
proves a tenantId:"system" spoof still bills/logs the real caller, a
cross-tenant spoof is blocked by FROM_NOT_OWNED against the real caller, the
credit gate applies to the real caller (not the bypassed "system"), a
user-permissions call with no tenant header 400s, and the legitimate
internal/system path is unaffected. Full comms suite: 439/439 passing.
C2. /billing/credits/verify let a buyer substitute a bigger pack than they paid for — FIXED
File: apps/manager/src/routes/billing.ts (POST /billing/credits/verify)
The Razorpay signature is HMAC-SHA256(order_id + "|" + payment_id, key_secret)
— it proves the (order, payment) pair is genuine, but binds nothing to
which pack was purchased. The handler read packId straight from the request
body:
const pack = await getCreditPackById(db, packId) // packId is client-suppliedExploit: create a Razorpay order for the small pack (₹415/500cr), pay it,
then call /credits/verify with packId: "large" (₹4,150/6,000cr+bonus) — the
large amount is credited. The authoritative payment.captured webhook (which
does read the pack from notes) is a deduped no-op on the same
rzp_pay_<id> idempotency key, so it can't correct the over-grant.
Fix:
- Added
RazorpayProvider.getOrder(orderId)(apps/manager/src/providers/razorpay.provider.ts) — fetches the order Razorpay actually recorded. /credits/verifynow ignores the client'spackIdentirely. It fetches the order, readsnotes.tenantId/notes.creditPackId(written server-side at/credits/buy), and:- Rejects (403) if the order's tenant doesn't match the caller.
- Rejects (404) if the pack no longer exists.
- Rejects (400) if the order's paid
amountdoesn't match the pack's current price (defends against tampered notes or a price change between buy and verify). - Only then credits
pack.credits + bonusto the verified pack.
Tests: apps/manager/test/credits-verify.test.ts — mock RazorpayProvider
extended with a getOrder backed by an in-memory order registry; all 8
existing tests updated to register their order first; 3 new SECURITY: tests
added (pack-substitution attempt credits only the real 500, not the claimed
6000+; cross-tenant order is 403'd; amount/pack mismatch is 400'd). Full
manager suite: 233/233 passing (was 230; +3 new).
🟠 HIGH — exploitable or real money/data harm
H1. Every /billing/internal/* cron endpoint was callable by any logged-in user — FIXED
File: apps/gateway/src/routes/manager.proxy.ts
Had no /internal block (the media and chatbot proxies both 404 their
/internal/*). The gateway stamps x-gateway-key on every proxied
request — the only thing the internal routes check — so any authenticated
user could trigger escalation, addon-renewals, recurring-charges, reconcile,
cleanup, or audit-integrity on demand: a cross-tenant stat leak, on-demand DB
load, and it made H6's non-idempotent renewal debit user-triggerable.
Fix: managerProxy.all("/billing/internal/*", (c) => c.json({error:"Not found"}, 404)),
registered before every other route so it wins regardless of declaration
order elsewhere — mirrors the existing media.proxy.ts / chatbot.proxy.ts
pattern of making the internal surface unroutable rather than merely
unauthenticated.
Tests: apps/gateway/test/manager.proxy.test.ts (new, 5 tests) — every
known internal route 404s with no auth, with a spoofed gateway key, and with
a bearer token; a sibling non-internal /billing/* route is unaffected. Full
gateway suite: 24/24 passing.
H2. POST /tenants farmed unlimited free credits — FIXED
File: apps/manager/src/routes/tenants.ts
Called provisionFreePlan(tx, tenantId) with the default grantCredits: true
and applied no owner cap or rate limit — unlike its sibling /workspaces,
which deliberately passes {grantCredits: false} and caps ownership at
FREE_WORKSPACE_LIMIT = 3 for exactly this reason. Looping this route minted
unlimited free (100cr-each) workspaces. Confirmed via a repo-wide search that
no frontend, CLI, or MCP surface actually calls POST /tenants (only its
PATCH/DELETE/transfer siblings are wired up) — the route was hardened
in place rather than removed, since a caller can't be fully ruled out.
Fix: FREE_WORKSPACE_LIMIT and a new shared countOwnedWorkspaces(db, userId)
helper were extracted from workspaces.ts (exported, and /workspaces itself
now calls the helper too — no logic duplicated). POST /tenants now checks
the same cap before creating anything (403 WORKSPACE_LIMIT_REACHED at the
limit) and calls provisionFreePlan(tx, tenantId, { grantCredits: false }) —
this route creates an additional workspace, never the true first-ever one
(that grant only ever happens once, via /onboarding/complete).
Tests: apps/manager/test/tenants-create.test.ts (new, 5 tests) — succeeds
under the cap with zero welcome credits granted; blocks at the cap with no
writes; looping the route can't exceed the cap or farm credits; membership in
someone else's workspace doesn't count against the caller; a soft-deleted
owned workspace frees a slot. Required widening the shared PGlite test schema
(test/setup.ts's tenants table gained slug/status/plan_id/logo_url/
database_config to match production). Full manager suite: 238/238 passing
(was 233; +5 new).
H3. Public chat widget could drain a tenant's balance with no rate limit — FIXED
File: apps/chat-engine/src/routes/widget.route.ts
apps/chat-engine/src/index.ts bypasses gateway auth for /chat/widget/*
(a public embed can't hold a JWT), and chat-engine's wrangler config declared
no rate-limit binding for it at all (unlike forms' 20/min-per-form+IP or
comms' SEND_LIMITER). The widget token is public by design — shipped in the
tenant's own page HTML — so a scripted loop had no throttle standing between
it and the workspace's entire credit balance.
Fix: added a native [[ratelimits]] binding (WIDGET_RATE_LIMITER, free,
no KV quota — 20 requests/60s) to apps/chat-engine/wrangler.toml, and a
check at the very top of the /message handler — before the widget token
even touches the DB, mirroring forms' "rate limit FIRST" convention. Keyed on
${widget_token}:${ip} together (not either alone): a token-only key would
let one abusive visitor lock out every other visitor of the same widget; an
IP-only key would let a botnet spread the same flood across many IPs. Fails
open (skips the check) when the binding or the IP header is unavailable,
matching every other fail-open convention in this codebase. A per-channel
daily cap was considered but deliberately left out of scope — this
codebase's established convention is native ratelimit bindings only, never
KV-based per-request counters (KV's free tier is 1,000 puts/day), and the
sustained-rate throttle alone already closes the acute drain vector.
Tests: apps/chat-engine/test/widget-rate-limit.test.ts (new, 6 tests,
using an in-memory fake limiter) — 429s past the limit without reaching the
DB; keys on token+IP together in both directions (different token same IP,
and same token different IP, are each independently limited); the exact key
shape is verified; fails open when the binding or the IP header is absent.
Full chat-engine suite: 132/132 passing.
H4. Integrity audit fired false CRITICALs daily and mis-reported the one real skew — FIXED
File: apps/manager/src/services/integrity-audit.ts
Check 2 summed only tx_status IN ('settled','held'), excluding
captured/refunded/voided — but those statuses only ever appear on a row
via an UPDATE ... SET tx_status = X, which never re-touches amount or
balance (confirmed by reading every txStatus: write site in ledger.ts:
rows are only ever inserted as settled or held; the other three statuses
are pure historical markers on existing rows, and every real compensating
movement — a capture delta, a void refund, a refund — is always a separate
new row already counted). Excluding a status-flipped row therefore dropped
real, permanent balance history from the sum. Verified against live prod
(2026-08-31) before fixing: the audit had filed a false ledger_sum_mismatch
critical every day since 2026-08-28 against a tenant whose books were
perfectly balanced (balance=346, real tx sum=346), reporting a phantom
ledger_sum: 390; for the one tenant with a genuine historical skew, it
reported −1470 instead of the true −1371. An alarm that cries wolf daily
is worse than no alarm.
Fix: removed the status filter — SUM(amount) over every row, unconditionally.
Tests: 3 new cases in apps/manager/test/integrity-audit.test.ts — a
healthy tenant stays clean through a hold→capture-with-delta-refund, a
hold→void, and a charge→refund. Full manager suite: 241/241 passing
(was 238; +3 new).
Live reconciliation for the one genuine skew: done — see the "Live prod-data findings" section below for the exact rows inserted. This needed explicit confirmation before writing to the production ledger, so it was tracked and executed separately from the code fix above.
H5. Abandoned checkout granted paid-tier limits with no payment — FIXED (bundled with the trial-inversion Medium bug, by explicit user request)
Files: apps/manager/src/routes/billing.ts (/checkout, /payment/verify, the webhook handler), packages/core-database/src/schema.ts
/checkout wrote planId: <paid>, status: "trialing" into the LIVE
subscriptions row before any payment happened; rebuildEntitlements
reads sub.planId and never consulted status. A free tenant who opened
checkout for Business and abandoned the tab kept a trialing row that any
later rebuildEntitlements trigger (buying an addon, the seat-gate
self-heal, an unrelated webhook) upgraded to full Business limits —
indefinitely, since nothing swept stale trialing rows.
Fixing this properly touches the exact same lines as a separately-flagged
Medium bug — trial eligibility was inverted: const trialDays = (existing?.hasUsedTrial || existing?.status === "active") ? 0 : ... zeroed
the trial for every real Free tenant (every Free row is also
status:"active", seeded that way by provisionFreePlan), while a tenant
auto-downgraded to status:"canceled" for non-payment could claim a fresh
trial on re-subscribe. Fixed both together (user's explicit choice, to avoid
touching this state machine twice).
Fix:
trialDaysnow gates onexisting?.hasUsedTrialalone — the flag that's actually supposed to mean "trial already consumed."/checkout's fresh-signup branch (previously the only one that wrote live columns) now stages its target inpending_*exactly like the upgrade flow already did, including a newpendingTrialEndcolumn (migration2026-08-31-pending-trial-end.sql) so a trial-eligible signup's end date isn't lost while waiting on payment. The idempotent-double-click dedup that used to fall out of the live write being immediate now has its own explicit check (§3c) keyed on the pending columns instead./payment/verify's promotion logic (isPendingUpgrade) now uniformly covers both a genuine plan-switch upgrade and a fresh signup — a genuine upgrade never setspendingTrialEnd, so the sameisTrialingcomputation naturally yields "no trial" for upgrades and the real trial window for a fresh signup, with no extra branching.- Regression caught before it shipped: fixing this exposed that the
subscription.charged/.activatedwebhooks look up a subscription only by its liverazorpaySubscriptionId— for a fresh signup whose id now lives only inpendingRazorpaySubscriptionIduntil verify promotes it, a webhook racing ahead of the frontend's/payment/verifycall (previously a safe, tested scenario) would drop asunknown_subscription. This is H9's already-documented gap, previously masked for fresh signups only because they used to write the live column immediately. Rather than accept a new regression, the webhook lookup was broadened to match onpendingRazorpaySubscriptionIdtoo, and the.activated/.chargedcases promote a pending match the same way/payment/verifydoes — closing H9 for these two event types specifically (H9 itself, for the other event types, remains open).
Tests: rewrote the two now-incorrect fresh-signup assertions in
billing-routes.test.ts (they asserted the vulnerable behavior), added a
SECURITY (H5) test proving rebuildEntitlements mid-checkout installs
Free's limits (not the paid plan's) before any payment; added 2 trial
regression tests to full-flows.test.ts (a genuine Free row gets its real
trial; a hasUsedTrial=true tenant gets none regardless of current status)
plus a webhook race test for .activated specifically (the realistic race
for a trial-eligible plan, since no .charged event exists until the trial
ends). Full manager suite: 245/245 passing (was 242; +3 new files'
worth spread across billing-routes.test.ts and full-flows.test.ts).
H6. Accumulator settlement race mis-bills (over- or under-charge) — FIXED
File: packages/core-billing/src/accumulator.ts (accumulateAndMaybeSettle, flushUnbilledUsage)
increment() does not bump settlementSeq, so two concurrent settles (two
chats, two emails) can read different accumulator values under the same
seq, compute different credit amounts, both attempt decrement() with
their own amount under a CAS on that shared seq — only one wins, and
whichever charge landed is not necessarily the one whose decrement won. Net
effect: 1 credit of usage can be silently dropped, or double-billed at the
next settlement. Affects chat (brain, chat-engine), email (comms), forms.
Fix: both settlement paths now skip their decrement() call whenever
charged.idempotent === true — only the caller whose ledger INSERT actually
won the race (guaranteeing its own creditsToCharge is the amount really
debited) ever decrements the accumulator; every losing/echoing concurrent
call returns a fresh store.read() instead of touching the store.
Bonus fix found while testing this (not separately numbered):
isUniqueViolation() in ledger.ts checked e.message ?? e.cause?.message
for the constraint name, but drizzle-orm wraps the raw driver error in a
DrizzleQueryError whose OWN .message is just the query text — ?? never
falls through to the real .cause.message, so it silently never matched a
genuine unique-index collision once the wrapper was in play. That turned
every idempotency-race loser (any concurrent charge/hold/adjust/grant
sharing a key) into an unhandled thrown error instead of a graceful
idempotent: true echo — undetected until this pass because no prior test
exercised two concurrent calls sharing a key with genuinely different
amounts. Fixed by preferring Postgres's own .constraint field (exact,
driver-stable) and checking the innermost .cause message first in the
fallback.
Tests: test/accumulator.test.ts — new race test seeding a 2-crossing
scenario under one unbumped seq, asserting the accumulator's reduction always
matches exactly what was actually charged regardless of which racer wins.
test/ledger.test.ts — new regression test for the isUniqueViolation fix
(two concurrent charges, same key, different amounts: both must resolve, one
real one idempotent, never an unhandled throw).
H7. capture() / void() / refund() didn't check their status-flip CAS — FIXED
File: packages/core-billing/src/ledger.ts
All three methods moved money via creditUnconditional, inserted their
compensating row, and only then flipped the hold's/original tx's txStatus
with a bare UPDATE ... WHERE txStatus='held'|'settled' — capture()/
void() had no .returning()/length check at all; refund() checked
but only after already crediting, so a lost race left an orphaned settled row
behind even after its own credit was undone. A concurrent capture+void on the
same hold — realistic because the only hold-expiry sweep left running is a
5-row opportunistic piggyback inside canChatProceed (the dedicated
brain-service cron for this is disabled pre-launch) — both read held, both
credit, and the hold is refunded twice. balance == SUM(tx) still holds
(both credits are real ledger rows), so H4's audit could never catch this
class of bug even after it's fixed.
Fix: all three now CAS-flip status FIRST (.returning() + length check)
and only move money if the flip won. A lost race throws HoldNotActiveError
(capture/void) or "already refunded" (refund) with zero money moved and
zero rows inserted — no compensating-undo step needed anymore, since a
loser never touches money in the first place.
Tests: two new genuine-concurrency tests in test/ledger.test.ts
(Promise.allSettled against real pglite, matching the file's existing race-
test pattern): concurrent capture+void on the same hold resolves exactly one
winner with exactly one settled row (a stray third row would mean the loser
also moved money); concurrent double-refund on the same tx resolves exactly
one winner with exactly one refund row (no orphan from the loser).
H8. Recurring-charge cancellation is a one-way door; resume is dead code — FIXED
Files: packages/core-billing/src/recurring.ts, apps/communication/src/index.ts (DELETE /v1/aliases)
registerRecurringCharge treated a row with status='cancelled' (and
'paused') as "already registered" and returned it as idempotent with
txId: null — no new charge, no reactivation. Repo-wide grep confirmed
resumeRecurringCharge had zero callers. Confirmed deterministic
scenario: release a vlozi.app alias and re-claim the same slug — the row
stayed cancelled/paused forever, the resource was refused at send time
(FROM_NOT_OWNED / MAILBOX_SUSPENDED), and there was no API path to repair
it short of a manual DB edit. Also, DELETE /v1/aliases never called
cancelRecurringCharge at all, so a released alias kept being billed
20cr/30d indefinitely, and a second tenant claiming the freed slug was billed
too.
Fix:
registerRecurringChargenow reactivates a cancelled/paused row instead of short-circuiting: a newreactivateRecurringChargehelper charges the current period (its own idempotency key,recurring:<id>:reactivate: <periodIndex>, distinct from both the original registration's key and the sweep's key) and CAS-advancesperiodIndex/statusback to active. Concurrent reactivations of the same row share that key (both read the same pre-reactivationperiodIndex), so a CAS loss just means another call already applied the identical charge — no refund needed or correct, unlike a brand-new registration's randomized-id race.- New
resumeAllPausedRecurringCharges(db, tenantId)givesresumeRecurringChargeits first caller — resumes every one of a tenant's paused rows (does not charge anything itself; the next sweep does, once the row is due again). DELETE /v1/aliasesnow callscancelRecurringChargefor the released slug (best-effort, wrapped so a billing misconfiguration can't turn a successful release into a 500).
Tests: packages/core-billing/test/recurring.test.ts — 7 new tests
covering reactivation of a cancelled row, of a paused row, a clean failure
on insufficient credits (row stays cancelled, no partial state), idempotent
retry of the same reactivation, concurrent reactivations charging exactly
once, and resumeAllPausedRecurringCharges resuming only paused rows
(active/cancelled untouched). apps/communication/test/aliases.test.ts's
existing release tests continue to pass unmodified.
H9. Webhooks never matched pending_razorpay_subscription_id — FIXED (as a required side effect of H5, not a separate pass)
apps/manager/src/routes/billing.ts webhook handler looked up
subscriptions.razorpaySubscriptionId (the live column) only. If a paid
upgrade's /payment/verify callback was lost (tab closed mid-checkout), the
.charged/.activated events for the pending subscription id were
dropped as unknown_subscription and permanently marked processed
(deduped) — customer billed for the new plan, local row never promoted, old
plan's Razorpay subscription never cancelled. This was originally scoped to
exactly these two event types, and it's exactly what H5's fix needed anyway
(a fresh signup's first webhook can now race the same way an upgrade's
always could) — fixed together rather than as a separate pass; see H5 above
for the implementation and tests. Original fix description, now
implemented: look up by razorpaySubscriptionId OR pendingRazorpaySubscriptionId; on a pending-column match for
.charged/.activated, run the same promotion block
/payment/verify uses.
H10. Refunds always landed in the permanent (non-expiring) bucket — FIXED
Files: packages/core-billing/src/ledger.ts, packages/core-database/src/schema.ts (new subscriptionPortion column on credit_transactions)
creditUnconditional (used by refund, void, and the capture delta)
always credited permanent_balance, by documented design ("don't re-apply
the original expiry"). But refunds aren't only failure-driven —
apps/blog-service/src/services/publish-billing.ts calls ledger.refund on
a user-initiated unschedule/delete-while-scheduled with no time bound. A
tenant could drain their expiring subscription credits into scheduled posts,
then unschedule after the subscription period ends — the refunded credits
landed as permanent, fully bypassing use-it-or-lose-it. The brain
knowledge-ingest hold path did this involuntarily on every ingest (refunding
the unused portion of every hold into permanent).
Fix:
- New nullable
credit_transactions.subscription_portioncolumn (migration2026-08-31-subscription-portion.sql) records how much of a debit's amount was drawn from the subscription bucket specifically. NULL on legacy rows — treated as "unknown, assume fully permanent" (the exact pre-fix behavior, so in-flight holds at deploy time keep working). - New
debitBuckets()helper captures the subscription/permanent split atomically with the debit (self-join onprev, same NOW() within one statement — no clock-skew window); wired intocharge(),hold(), andadjust()'s negative branch, which now persist it on their tx-row insert. - New
creditBackToOriginBuckets()helper returns money to the SAME buckets a debit drew from, LIFO: since a debit drains subscription first then permanent, a partial credit-back (e.g. a capture's unused delta) returns the LAST-drained dollars (permanent) first, only spilling into subscription once the permanent-drawn portion is fully restored. A subscription credit-back landing after the period has since elapsed falls back to permanent instead of resurrecting a dead bucket.capture(),void(), andrefund()all now use this instead ofcreditUnconditional.
Tests: rewrote the 3 test/ledger.test.ts bucket-semantics tests that
had asserted the OLD "always dumps into permanent" behavior (that was
precisely the bug), plus 3 new tests: refund falling back to permanent once
the subscription has since elapsed, LIFO partial credit-back across a
mixed-bucket hold, and a full refund of a mixed-bucket debit restoring the
exact original split.
H11. MCP API keys bypassed the Settings → Services on/off toggle — FIXED
Files: packages/mcp-core/src/entitlements.ts (new), packages/mcp-dispatch/src/index.ts, apps/mcp-gateway/src/{server,index}.ts, apps/flows-service/src/{routes/flows,engine/run-engine}.ts
The REST API-key path enforces getUserDisabledServices
(apps/gateway/src/middleware/api-key.middleware.ts); the MCP auth path
(packages/mcp-core/src/auth.ts, packages/mcp-dispatch/src/index.ts)
checked permission + plan only. An owner who disabled e.g. "newsletter" to
stop spend still had an MCP-key-holding agent able to call
newsletter.send_campaign and be charged.
Fix:
- New
getMcpDisabledServicesinpackages/mcp-core— a port of the gateway'sgetUserDisabledServices(sameenabled && !userDisabledrule, same fail-open-on-DB-trouble contract, 60s cache) but built oncreateCachedHttpDb(neon-http) rather than gateway's Pool-based client: this runs on everytools/listand everytools/call, a genuinely hot path, and a Pool-based client must never be cached across requests in a Worker. mcp-dispatch'slistTools/dispatchToolnow check it. ANamespace -> service_codemap handles the two real name mismatches (chat->chatbot,flow->flows) and the two namespaces with no toggleable service at all (billing,workspace— always allowed). Newservice_disabledMcpErrorCode, mapped to 403 in both ofmcp-gateway's error-handling paths (JSON-RPC and the REST shim).listToolsis now async (reads the toggle state) — every call site updated (mcp-gateway's twotools/list/GET /toolshandlers,flows-service's twodispatchToolcall sites now threadcoreDatabaseUrlthrough).
Tests: packages/mcp-core/test/entitlements.test.ts (new, 6 tests) —
fail-open on no URL, both disable conditions, the "not disabled" case, TTL
caching, fail-open on a DB error. packages/mcp-dispatch/test/ service-toggle-gating.test.ts (new, 8 tests) — hides a disabled namespace's
tools from listTools while leaving others visible, never blocks
billing/workspace even if their name somehow appeared in a disabled set,
dispatchTool rejects a disabled namespace without touching the network,
allows an unaffected namespace, and permission/plan checks still run (and
short-circuit) before the entitlement check.
🟡 MEDIUM (condensed; all FIXED 2026-09-01)
Fixed in one coordinated 15-PR pass (see PR-N refs below — commits on
main, this session). Every item got its own regression test; the four
touching real money movement (PR-8, PR-9's underlying data, PR-10, PR-11,
PR-12, PR-13) got genuinely new tests, not just "existing tests still pass."
- Downgrade-to-Free writes
status:"canceled"— FIXED (PR-12). New"downgraded"status:autoDowngrade()now writes it instead, and the gateway's subscription guard passes it through as free-tier access (same as a tenant who never paid), while"canceled"keeps hard-blocking. Needed a coordinated gateway+manager deploy. Also extended internal-api's revenue report with a separatedowngradedlist so these tenants don't silently vanish from ops' churn view. Trial logic is inverted.FIXED alongside H5 above (bundled by explicit user request, since both bugs lived in the same lines).- Yearly→monthly plan switch collapses prepaid expiry — FIXED
(PR-13b).
creditSubscription'ssubscriptionExpiresAtwrite is nowGREATEST(stored, new)instead of an unconditional overwrite. processAddonRenewalsisn't concurrent-sweep-safe — FIXED (PR-8). Added an idempotency key (addon_renewal:<id>:<nextRenewal>) to the renewal debit and a CAS on thenextRenewaladvance, mirroringprocessRecurringCharges.- Past-due state machine may never trigger — FIXED (PR-6c),
confirmed via live Razorpay docs research (not guessed):
payment.failedandsubscription.pendingare the same underlying active→pending transition; both now share the existing past-due-stamping case instead of one being dropped as an unhandled event. - No notification when an addon/alias/mailbox is paused — FIXED
(PR-9). Both sweeps now best-effort email the workspace owner
(
notifyResourcePaused) with a human-readable resource label; a notification failure never fails the sweep's own response. - Webhook handler returns 200 on unexpected processing errors — FIXED (PR-6a). Now returns 500, so Razorpay retries instead of a silent permanent drop; every path here is already idempotency-keyed, so a retry is safe.
grant()is move-first, undo can drive a bucket negative — FIXED (PR-13a). The permanent-bucket undo is now floored at 0 (GREATEST), with a warning logged when the clamp actually fires.- Fail-open billing in collections/forms with no observability — FIXED
(PR-3b).
db.middleware.tsnow warns whenCORE_DATABASE_URLis unset; thewaitUntil(meterSubmission(...))call now logs (tenant/form context) instead of silently swallowing a metering failure. - Newsletter
fireCampaigncharge-without-work window — FIXED (PR-10). The refund-guarded region now starts right after the charge succeeds instead of partway through the send pipeline, so a throw anywhere in between refunds and rolls the campaign back to draft. - Content-engine DLQ/interrupted paths keep the charge without refund —
FIXED (PR-11).
CreditLedger.findByIdempotencyKeyis now public; the interrupted-run guard looks up the original charge by its deterministic key and refunds it before marking the target failed, with a double-refund-safe no-op on a second interrupted redelivery. - No hold-expiry sweep is actually scheduled — RE-CONFIRMED, still intentionally disabled (PR-14a). Re-checked 2026-09-01: still pre-launch (marketing site targeting ~mid-Sept 2026), so the brain-service cron stays commented out rather than being flipped blindly — re-verify tenant volume at actual launch.
- Accumulator flush functions exist but are never scheduled — FIXED
(PR-14b). Added a "list tenants with >= 1 whole credit outstanding"
query per accumulator, a new manager route (chat/email) and a new
forms-service route (forms' own regional DB), wired into
billing-crons' daily job list via a new per-job service binding.
⚪ LOW / cleanup (all FIXED 2026-09-01)
- Cross-tenant unbilled outreach scan in contact-intelligence's
trigger_outreach_scan— FIXED (PR-2).runOutreachScannow takes an optionaltenantId; both HTTP-reachable entry points scope to the caller's tenant, while the genuine 6h all-tenants cron path stays unscoped by design (a real cross-tenant leak, despite its Low label — sequenced early in the plan for that reason). - AI chat cost pre-flight estimate excludes system prompt/history/RAG —
FIXED (PR-15). A documented static
AI_CHAT_SYSTEM_PROMPT_OVERHEAD_TOKENSis now added on top of the raw message at gate time for chat-engine's two entry points and brain-service's owner-chat route (reordering to build the real prompt first was rejected — it's fused with a CI-context/RAG/intent pipeline the gate can't afford to run twice). Brain-service's customer route, which already builds the full prompt before gating, is untouched. - Admin credit-adjust idempotency key is
Date.now()-based — FIXED (PR-5b).adminAdjustSchemanow accepts an optional caller-suppliedidempotencyKey, threaded through both the grant and revoke branches (revoke previously had no protection at all). buyAddonquantity has no upper bound — FIXED (PR-5c). Capped at 100, enforced at both the HTTP schema and inside the service function (defense-in-depth against a future internal caller).- Razorpay HMAC comparison is not constant-time — FIXED (PR-5a).
verifyWebhookSignature(andverifyPaymentSignature, which delegates to it) now use the existingtimingSafeEqualhelper instead of===. - Reconciliation treats "subscription already cancelled" as a failure —
FIXED (PR-7).
cancel_razorpay_subnow checks the live status first and treats already-cancelled as success, instead of retrying an operation that can never succeed into an abandoned-alert page. subscription.cancelleddoesn't clearpending_*/pastDueSincelike.halteddoes — FIXED (PR-6b). Collapsed into one UPDATE matching.halted's exact shape (confirmed.halteddoesn't cleartrialEndeither, correcting the audit's original wording).tenants.planIdnever kept in sync withsubscriptions.planId— FIXED (PR-4). Deprecated rather than synced (user's explicit choice): internal-api's admin analytics now readssubscriptions.planIdeverywhere, including a previously-missingsubscriptionsjoin in the revenue plan-distribution query.- Stale prices in
public-docs/— FIXED (PR-1). "1 credit per platform/account per day" corrected to 2, matchingCONTENT_PUBLISH_PER_PLATFORM/ANALYTICS_PER_ACCOUNT_DAILY. content.*MCP tools charged but nobillingmetadata in the service catalog — FIXED (PR-3a). Addedcontent.schedule_content's billing metadata (the catalog's firstperUnitentry) and its expected-tools parity-test coverage.- Unbounded growth of
processed_payment_events/billing_signature_failures— FIXED (PR-3c)./billing/internal/cleanupnow also prunes both (>90 days / >180 days respectively).
📊 Live prod-data findings (read-only queries, 2026-08-31)
-
All core invariants hold across the 6 live tenants: no negative balances, no duplicate idempotency keys, no expired-but-still-pending holds,
recurring_chargesrows are healthy (comms + mailbox, bothactivewith sanenext_charge_at). -
One tenant carried a genuine −1,371 skew (
ls_b8db55b0-..., i.e.balance(522) − SUM(tx)(1893) = −1371) — RECONCILED 2026-08-31. Its full ledger history had a jump where a row's recordedbalance_after(591) didn't follow from the previous row'sbalance_after(2066) plus its ownamount(−4) — a silent, unaudited reduction of exactly 1,471 happened between them. That matches the pre-fix ledger zeroing an expired subscription bucket without writing thesubscription_expiredaudit row the current code now writes (seeexpireElapsedWithAuditinledger.ts, closed this session). Separately, the tenant's first-ever ledger row already assumed a starting balance with no grant row behind it — its 100 welcome credits never got aprovision.free_planrow (the general-purposebackfill-free-plan-tx.mjsscript only catches positive-skew tenants; here the +100 gap was masked inside a much larger negative-direction gap, so that script's own filter couldn't see it). The two gaps net to the measured skew:+100 − 1471 = −1371.Reconciled with the user's explicit confirmation (a direct write to the production ledger, held separate from the code fixes) by inserting two idempotency-keyed rows into
credit_transactions, both stampedbalance_after = 522(today's real balance, matchingbackfill-free-plan-tx.mjs's own convention of not trying to reconstruct historical balance_after values):+100reasonprovision.free_plan, keyfree_welcome_<tenantId>;−1471reasonsubscription_expired, keyreconcile_sub_expired_<tenantId>. The tenant's spendablebalancewas never touched (522 before and after) — only the audit trail was completed. Verified post-reconciliation:balance(522) == SUM(tx)(522), skew0, and re-running the fixed H4 query against all 6 live tenants finds zero mismatches. -
This −1,371 skew was exactly what H4's integrity-audit bug was mis-reporting as
−1470before either fix — H4's code fix makes the audit report the true number, and the reconciliation above makes that number0rows above.
What's solid (verified by reading, across all three reviews)
- Debit atomicity: every charge/hold/negative-adjust is a single
UPDATE ... WHERE effective_balance >= amountCAS statement with inline subscription-expiry zeroing — concurrent overspend is structurally impossible, no retry loop needed. - Per-tenant idempotency via a partial unique index on
(tenant_id, idempotency_key);refund()correctly nulls the key on success so a charge→refund→retry cycle re-charges rather than replaying the refund. - Tenant identity at the edge is sound everywhere except C1: the gateway
deletes every spoofable identity header before re-stamping from the
verified JWT/API key, every service gates on
x-gateway-key, and public endpoints (forms, collections, chat widget, webhooks) derive tenant from a DB row rather than the request. - The blog publish live-cycle billing (pre-read → charge → CAS on exact status → rollback on lost race) is the most carefully-built billing flow in the codebase; REST/MCP parity is exact.
- The recurring-charge registration race (two concurrent fresh claims)
is correctly closed via
onConflictDoNothing+ loser self-refund — H8 is about re-registering a cancelled resource, a different code path. - Seat-gate enforcement (shipped this session) correctly counts members +
pending unexpired invites, self-heals a missing
tenant_servicesrow, and every membership-removing path frees the seat. no-direct-writes.test.tsenforces that nothing outsidepackages/core-billingwrites the three credit tables directly.- The addon-lifecycle hardening (shipped this session) is transactional, idempotency-keyed, and correctly snapshots the purchase-time price — the only gap found in this audit (H6-adjacent: no CAS on the renewal advance) is new information, not a regression.
Recommended fix order for what remains open
H1, H2, H3— done (2026-08-31): all three were pure access-control gaps (missing gateway block, missing rate cap, missing rate limit), each fixed as a small, isolated change with new regression tests.H4 + the live −1,371 reconciliation— done (2026-08-31): the audit's ledger-sum SQL now sums every row regardless of status, and the one tenant's genuine skew was reconciled with two idempotency-keyed backfill rows (with explicit confirmation before the ledger write). All 6 live tenants now pass the fixed check with zero mismatches.H5— done (2026-08-31), bundled with the trial-inversion Medium bug by explicit user request. Also closed H9 for the two event types this fix's own regression check surfaced (subscription.charged/.activated); H9 remains open for the other event types.H6 – H8, H10, H11— done (2026-08-31), one coordinated pass throughpackages/core-billing(accumulator/ledger/recurring),packages/ mcp-core+packages/mcp-dispatch(new shared entitlements module + namespace mapping),apps/mcp-gateway+apps/flows-service(wiring), andapps/communication(alias-release cancel call), by explicit user request ("one coordinated pass, all 5 together"). Also fixed a previously-undetected bug inisUniqueViolation()found while testing H6 (see H6's writeup) that had been silently turning every idempotency-race loser acrosscharge/hold/adjust/grantinto an unhandled thrown error once drizzle-orm's error wrapper was in play. Every touched package/app typechecks clean and its full test suite passes:core-billing136/136,manager245/245,communication439/439,blog-service/newsletter-service/content-engine/brain-service/chat-enginefull suites,mcp-core35/35,mcp-dispatch20/20,mcp-gateway37/37,flows-service47/47.Mediums and Lows— done (2026-09-01), one coordinated 15-PR pass (PR-1throughPR-15), each independently committed tomainwith its own typecheck + targeted tests + full-suite run. None of these commits have been pushed or deployed yet — that's a separate step, not yet requested.PR-12(downgrade-status split) specifically needs a coordinated gateway+manager deploy — shipping either side alone creates a window of either hard-blocking or over-granting access to nonpayment-downgraded tenants.apps/billing-crons/wrangler.toml's newFORMS_SERVICEbinding (PR-14b) also isn't committed — that file isn't tracked in git (unlike its 10 sibling services), which needs a deliberate decision before deploy, not just a push.