Every AI surface follows the same four rules: show the grounding, expose confidence, require human approval before customer contact, and capture accept/edit/reject feedback that flows to the model console (W28). All generation runs on the local purpose-tuned LLM (3.3) unless a task is explicitly switched to a hosted API through the adapter interface. v4.3 adds the model lifecycle pipeline (W29) - how an adapter graduates from eval to production with a human signing the promotion - and interaction review (W35), which turns each call, email, and visit into playbook-fit coaching and tracked commitments. Deliberately not built: a general-purpose chatbot (every surface here is task-shaped) and a standalone lead-scoring screen (the score lives on W19; its evaluation lives on W28).
W23 - AI composer (in-context drafting)
Drafting from inside a job or Contact 360: template intents, tone/length controls, personality match applied, and a grounding panel showing exactly which retrieved facts (pgvector) the draft used, plus consent and pricing-floor checks. The AI never sends - approve/edit/reject is the human gate, and that feedback trains the next adapter (W28).
Draft email - Dana Harmonlocal LLM · adapter v12
Follow-upFinancing infoSchedule ask
ToneFactualWarmLengthShortDetailed Regenerate
Subject: Your underlayment question + itemized financing breakdown
Hi Dana,
Following up on your question about the synthetic underlayment - attached is the spec sheet, including the valley ice-and-water detail we discussed on the quote. Since you were comparing monthly options, the Better option at $18,750 works out to $228/mo on the 84-month plan - the full amortization table is attached, no hard credit pull to see terms.
No rush on a decision. Happy to answer anything else line by line.
Jordan · Summit Roofing
Approve & open in editor Edit RejectAI never sends - a person always does · feedback trains adapter v13
Grounded on
Quote Q&A: underlayment vs. felt question (Jul 14)
Extends the W4 canvas with AI node types - generate message, classify reply, extract fields - under guardrails: a confidence branch routes anything below threshold to a human task, an approval-gate node is required before customer-facing sends, and each AI node has a model selector exposing the local-adapter vs. hosted-API choice from 3.3.
Automation: reply handlerLive
Run history+ Node
TRIGGER
Reply received
SMS or email
→
AI · CLASSIFY
Intent + opt-out
local · adapter v12
→
BRANCH
Confidence ≥ 80%?
else → human task
→
AI · GENERATE
Reply draft
personality-matched
→
GATE · REQUIRED
Human approval
before any send
→
ACTION
Send reply
Branch: opt-out → suppression list (no gate needed - always immediate)Branch: <80% → task for rep
Vision AI over the media hub (W20): proposed hail-hit annotations with per-detection confidence, accept/adjust/reject controls, and a certification path into the insurance packet. The principle: AI proposes, a named inspector certifies - nothing reaches an adjuster without human sign-off, and the packet records who approved what. Accept rates feed the model console (W28).
Inbound texts/emails auto-classified (interested · scheduling · question · not-now · opt-out) feeding the automation engine. The compliance hook is the point: AI-detected opt-outs write to the suppression layer immediately with no approval gate, while low-confidence classifications route to a human queue. Override rates feed W28.
Inbox - 23 newAI triage on
AllNeeds review (3)Interested (6)
Wu, K. - "Yes let's do Thursday morning if the crew…"scheduling · 97%→ booked via automation
Pratt, S. - "STOP texting me"opt-out · 99%→ suppressed · all channels
Harmon, D. - "Does the 84-month plan have a prepayment…"question · 94%→ draft ready (W23)
Unknown # - "my brother said you did his roof, we're at the lake house…"62% - human queue
Bell, R. - "Not this year, check back in spring"not-now · 91%→ nurture Mar 2027
Opt-outs today: 2
Written to suppression instantly - no approval gate on opt-outs, ever. Synced to CRM + data engine.
Last 30 days
Auto-handled78%
Human queue22%
Overrides11% ↑
W27 - Ask-your-data analytics
Natural-language questions over the warehouse, answered with the chart plus provenance - which continuous aggregate, which filters, the generated SQL - so answers are auditable, not oracular. Read-only access against Timescale aggregates; no AI write path to data.
Ask your datalocal LLM · read-only
cost per signed job for the June hail mail campaign, by weekAsk
Cost per signed job - Jun hail mail
Overall: $201 per signed job ($3,410 / 17 signed)
$412Wk of Jun 16
$247Jun 23
$168Jun 30
$121Jul 7
Falls over time as mail-piece-2 responses convert · suggested follow-up: "compare to Meta ads campaign"
The governance screen the v4 LLM decision demands: active base model and LoRA adapter version, held-out rubric scores side-by-side against the hosted-API baseline (the comparison the adapter escape hatch enables), GPU cost vs. estimated API spend, per-surface human feedback rates (W23/W25/W26), drift alerts, and the retune queue with rollback targets.
Model ops - AI layerServing healthy
Eval history Queue retune (v13)
Base model
Qwen2.5-14B
Apache 2.0 · adapter v12 (LoRA)
Serving
vLLM · 2× L4
p95 1.9s · 99.7% uptime
GPU cost / mo
$1,140
vs est. hosted API $3,400
Escape hatch
Adapter I/F
hosted API swappable per task
Eval rubric - local v12 vs hosted baseline (held-out set, blind-scored)
Playbook accuracy
8.8 /9.1Draft usability
9.0 /8.9Reply classification F1
.94 /.92Open-ended reasoning
7.1 /9.0
▮ local · | hosted baseline - open-ended gap accepted: no product surface depends on it (tasks are narrow & tuned)
Human feedback (30d)
Composer accept / edit / reject64 / 29 / 7%
Damage detection accept87%
Inbox triage overrides11% ↑
Triage override drift > 10% - retune suggested
Adapter registry
v12 · activeJun 30
v11 · rollback targetJun 2
v13 · training queuepending
W29 - Model lifecycle & promotion pipeline
How a model change ships: offline eval against the rubric → shadow mode beside the live adapter (output never shown to users) → canary A/B through the experiments console (W30) → promotion behind a required human sign-off → post-promote monitoring with a warm rollback target. No stage can be skipped, including for hosted-API swaps through the adapter interface; accepted/edited/rejected outputs from W23/W25/W26 become the next version's training set.
Model lifecycle - adapter v13 (candidate)stage 2 of 5 · shadow
Training data diff Rollback serving to v11
1 · OFFLINE EVAL
Passed
Rubric ≥ v12 on all four axes · triage F1 .96 (+.02) · held-out set, blind-scored
2 · SHADOW · RUNNING
Day 9 of 14
Runs beside v12 on live traffic, output never shown · agreement 94% · disagreements sampled for review (31 open)
3 · CANARY A/B
Queued
10% of composer traffic via Experiments (W30) · metric: accept rate · auto-stop on reject spike
4 · PROMOTION · GATE
Human sign-off required
Eval + canary results attached · sign-off: ops lead · promotion is logged, attributable, reversible
No stage can be skipped - including for hosted-API swaps via the adapter interfaceFeedback from W23/W25/W26 becomes v14's training set
W35 - Interaction review & relationship health
Every interaction - call, email, text, site visit - reviewed by the local LLM against the customer's personality playbook and stored history: playbook-fit checks, sentiment, and detected commitments tracked to completion (a promise made on a call becomes a task). Aggregates roll up to a relationship-health score on the customer side and a coaching view on the rep side. Guardrails are part of the design: per-state recording consent with played disclosure, reps see their own reviews first, manager views lead with aggregates, reviews coach and never auto-discipline, and disputes route to a human.
Interaction review - Dana Harmonlocal LLM · playbook C-Analytical
AYO & Ryzo Platform - hypothetical product specification and wireframe set, v4.0 (July 2026). Feature inventory derived from public JobNimbus and SumoQuote materials; all screens, names, and figures are illustrative. Back to index