Model slots wave (2026-09-20), chunk B:
- settings: new `models` section {strong, cheap, vision} slots (each
field independent, same shape as a role override); ROLE_SLOT maps all
nine roles to tiers (strong=prober+teacher, cheap=researcher/
demoBuilder/extractor/auditor/labeler, vision=visualMaker+pdfReader);
resolution chain per role: override → slot → factory pin.
- config: ModelSpec gains provider (all pins openrouter, the BYOK
default); MODEL_PINS in stores.ts now spans all nine roles (the strong
slot's factory rep IS the prober pin).
- engine: piModels() registers EVERY builtin provider catalog;
piEngineForSpec(resolved spec) is the new engine factory (pi-ai
resolves per-provider keys from env itself); piEngineFor stays as the
config-pin fallback. chat-handler resolves the conversation chain
(chat override → strong slot → pin) and injects engineFor into the job
runner, closing the old gap where per-role overrides never reached
job engines.
- /api/models: models federate ONLY keyed providers (providers surface
with keyPresent, env-wins, fresh per call); slots (resolved tier pins)
join rolePins; catalogDefaultsFor inherits the strong slot; the models
settings section gets the same catalog gate as chat.
- pricing: PRICES/CONTEXT_LIMITS uniformly keyed provider:id
(regenerated from ALL pi-ai catalogs, 1354 keys); usage stamps
(TurnUsage, JobUsage, extraction stages) switch to provider:id —
owner decision: no legacy compat, old stamps render unpriced.
Settings interlude (owner request, 2026-09-19): the route now also returns
rolePins: Record<BackgroundModelRole, {model: {provider, id},
thinkingLevel}> — the six background roles' RESOLVED models (the user's
agent-models override falling back to the config.ts MODELS pin), so the
settings UI shows each role row's 'Default: <actual pinned model>' instead
of a generic config caption. Same resolution the agent factories use
(effectiveSettings), so the caption can never drift from what runs.
Pure projection (rolePinsFor) + the fs wrapper (catalogRolePins); pins
tested against config defaults and model/thinkingLevel-only overrides.
BACKGROUND_MODEL_PINS exported from stores.ts for the test (was local).
- Cluster namer bills the money channel now: per-step usage + model
captured via onStep, persisted as NamingRunUsage records to
data/graph/naming-runs.jsonl (idempotent per startedAt; failed/aborted
runs persist too — money spent either way). callUsage promoted to
extraction.ts module scope for reuse.
- core/cost-summary.ts: the pure fold behind the pill — chat runs,
notes-source runs, and naming runs into stage totals (extraction /
audit / naming) + newest-first recent runs. Dollars INFERRED read-time
via prices.ts, never stored; unpriced models excluded-but-counted
(render '—', never a silent zero); tokens always sum.
- GET /api/graph/costs (src/server graphCosts composition, thin route).
- Graph view: bottom-left cost pill (cumulative dollars headline) +
expandable breakdown panel (stage rows with tokens secondary, recent
runs with time/title/merged/cost); fetched on mount, refetched when a
pipeline run finishes; merged into one bottom-left stack with the
running-pipeline pills.
- Tests: pure-fold suite, naming capture wiring (full importNotes ->
naming path), store idempotency + strict reads.
Live-smoke follow-up the user approved (2026-09-19): the GEB read showed
"78 pages consulted" when the reader genuinely READ ~12 - pdf_search hit
snippets inflated the count. PdfResult now carries both:
- pagesRead: extract/render output entered the loop - the honest count
the badge and brief footer show ("read N pages", "Pages read: ...").
- pagesConsulted: every page whose content entered the loop, search-hit
context lines included - unchanged, still the citation audit set (the
reader's instructions still say "Pages consulted: N, N, ..." and the
footer still lists it when it exceeds the read set).
The unread-finish gate deliberately stays on CONSULTED: search hits are
genuine consultation (anti-fabrication duty), so a search-only brief is
honest and must deliver - pagesRead is a reporting refinement, not a
delivery gate. Pinned by the search-only faux test (read [], consulted
[1], delivered).
UI tolerates pre-split persisted records via ?? fallbacks (old steers
lack pagesRead).
Learner source files, on the teacher's own terms (nothing auto-injects):
- read tool (src/agent/tools/read.ts): pi-read port — path/offset/limit,
2000-line/50KB truncation with use-offset continuation notices, image
files (jpg/png/gif/webp/bmp) returned as native image content parts
(new LhToolImageResult escape hatch in lh-tool.ts). Always-on in both
the prober and teacher registries (user-decided 2026-09).
- pdf-reader sub-agent (JobKind 'pdf', toggleable, ack-and-steer): triage
via pdf_info (pdfjs text layer: per-page text/image/math-density),
targeted pdf_extract, regex pdf_search (hits count as consultation),
and pdf_render — pages painted by pdfjs onto a canvas shim in headless
Chrome (the pdfium viewer doesn't paint headless), screenshot fed back
image-in-the-loop (visual-maker idiom, providerSafePng applied). The
unread-finish gate bounces a zero-consultation brief once. Teacher gets
a text brief with page citations only; renders never leave the loop.
- MODELS.pdfReader = qwen/qwen3.8-flash (user override 2026-09-19: the
look-at-renders workload wants the vision-proven model; catalog listed
glm-5.3-flash as image-capable but qwen is the known-good daily driver).
- UI: PdfCompleteBadge + PdfBriefCard (steer-card, job-card), dispatch
card + activity-dock kind entries (type-forced, flagged for UI review).
- Prompts: 'Learner source files' section in teacher/30-jobs.md; read
line in prober/10-probe-doctrine.md. Doctrine rides tool descriptions
per the self-contained-tools principle.
11 commits on fbeca36: minimal-color rounds 1-2 (phase color retired,
the braille spinner is the one working indicator; the chip is label-only),
typography rounds 1-5 (three-font rule enforced — CMU titles, Source Sans
session titles, JetBrains mono; snapped type ladder in globals.css with
the xxs rung merged into xs; lh-input caret at the font-metric floor;
session titles at prose size, message counts and active-row weight dropped,
ISO dates), header rounds 6/6b/6c/6d (60px top bar aligned to the sidebar
LH band; the open-sidebar toggle matches the close glyph at size-4 both
ways; SidebarToggle always-mounted with a lockstep animated slot — no more
one-frame title jump; title at text-3xl).
Semantic change folded in: `busy` in the chats list now means exactly "a
teach turn is in flight" — background jobs no longer spin the session row
(the sidebar's Background agents section is their channel).
Merge fallout fixed in the merge: components/graph-coverage.tsx (main's
coverage panel) held the last three text-xxs uses; with the rung deleted
they silently lost their size — mechanically swapped to text-xs, matching
lh-ui's own swap in graph-context-marker.tsx.
No file overlap with main's waves since fbeca36 (extraction coverage,
lh-graph merge, world-scale bake) — clean auto-merge. Gates: typecheck,
eslint, 496/496 tests, build. Live-verified on the dev server: 60px
header with 32px CMU title, label-only chip, snapped ladder active
(--lh-prose-size 16px, no --text-xxs rule anywhere), sidebar rows with
ISO dates, animated sidebar toggle with no header jump.
THE MESSAGE IS THE UNIT OF COVERAGE (vision-extraction-coverage.md): a
per-chat append-only ledger (coverage.jsonl) records which messages the
pipeline has extracted, by content fingerprint over exactly what the
extraction view sees (one shared line builder — core/coverage.ts); the
uncovered remainder is DERIVED at read time, never dual-tracked. A quiz
answer merging into the asking message re-opens only that message;
plan-block data-part attach never re-opens anything.
- The trigger (turn-service) replaces every-3-quiz-answers with
uncovered-tokens >= EXTRACTION.chunkTokens (4096) — quiz-less sessions
extract too, content size is the honest signal.
- runExtraction is now chunked: token-budgeted batches of the oldest
uncovered consecutive messages (a tool call + result never split), each
merged batch checkpointing its messages' coverage atomically — an abort
or error loses at most the in-flight batch; merged batches stay covered.
- Observation refs are batch-tagged (obs:<batchTag>:<sourceId>): re-runs
of the same batch dedupe; a re-opened message's fresh understanding
lands as a new event. Fixes the old obs:p1 position-collision.
- New API: GET /api/graph/coverage (exact per-session derived coverage +
running flags, most-uncovered first) and POST /api/graph/coverage/extract
(manual run for one session, 409 while in flight).
- Graph view chrome: the coverage button opens a panel with every
session's covered/uncovered tokens and a per-session Extract button —
the manual answer to aborted runs, errors, and dormant tails. Renders
even when the graph is empty: an empty graph with uncovered sessions is
exactly when it matters. Polls only while open.
- Notes imports keep the whole-file checkpoint path (coverage undefined).
- Tests: pure-core coverage suite (11), chunked-pipeline suite (3:
multi-batch + no-op, abort-keeps-earlier-batches + remainder re-run,
re-open alone + data-part invisibility), panel render suite (5);
pendingQuizAnswers and its tests deleted as dead code. 486/486.
Live-smoked on real GLM: manual extract of "teach me modal realism" ran
one batch, wrote 4 coverage entries, panel flipped to "1.3k covered ·
complete" with the button disabled, graph refetched (nodes 11), usage
persisted with real stats.
Nothing exists purely for decoration: phase color-coding is retired
app-wide (phases only advance — every session ends Teaching — so
phase-as-signal was decoration with a one-time function), and what
state genuinely deserved — 'a turn is in flight' — is now carried by
MOTION alone, in the terminal's own glyph language.
- BrailleSpinner (components/braille-spinner.tsx): the canonical
cli-spinner sequence (⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏) as exact dot bitmaps in a 2×4
braille cell, 80ms/frame (800ms cycle). Pure CSS: one shared keyframes,
frames staggered via animation-delay; no JS timer, no re-render per
frame — cheaper than the old breathing dot. Static contexts (SSR,
prefers-reduced-motion) show frame 0 — a legible glyph, not a blank.
- Sidebar rows: spinner ONLY while a teach turn is in flight; idle and
question-pending render NOTHING (a resting state has no scan value —
the meta line's 'Waiting for your answer' text carries it). The
three-state phase dot (pulse/solid/hollow) and PHASE_FILL/RING maps
are gone.
- Top bar PhaseChip: label-only pill; the old dot slot becomes the
spinner's home while a turn is in flight (busy prop wired from
ChatView's busy = localStream || turnInFlight). Transcript
PhaseDividers drop phase colors for monochrome rules + labels.
- API narrowing (app/api/chats/route.ts + ChatListItem contract):
busy = isTurnRunning ONLY — background jobs no longer count (the
sidebar's Background agents section is their channel; a per-row
spinner would double-signal them).
- globals.css: --phase-* tokens + lh-dot-breathe removed (dead after
the sweep — no remaining users).
Live-verified: idle/pending rows render no indicator, /chat/new chip is
label-only, 10 CSSAnimations at 800ms with correct 80ms stagger, static
frame 0 under reduced motion. Gates: typecheck, lint, 470/470, build.
Lane crossings to flag at merge: app/api/chats/route.ts + the
ChatListItem contract comment (src/core) — the busy-semantics narrowing;
components/chat.tsx one-line busy prop.
The continuous 0-4096 dial was precision theater (a 128-token budget is
~2 propositions; nobody wants that granularity). Decided stops (owner,
2026-09-15): 0 / 64 / 256 / 1024 / 4096 / 16384 — the native range walks
positions 0-5 and maps through GRAPH_CONTEXT_TOKENS_STOPS in contracts.
Max rises to 16384 (route range follows); the server still accepts any
integer in range so a stale client cannot brick settings; non-stop
legacy values snap to the nearest stop for the thumb, readout stays
honest.
One shot per chat, fired at the first turn's onFinish: the session's
topic (its only typed message) is embedded as an EmbeddingGemma retrieval
query; the semantically closest graph propositions are selected under the
session's token budget (pure budget, no similarity threshold — the
decided doctrine) and delivered as a marker-only steer through the wake
pipeline, queuing behind pending questions like every sub-agent result.
- contracts: ChatSettings.graphContextTokens (default 1024, max 4096,
step 128, 0 = off); GraphContextData + graphContextText; the
graph-context JobKind + buildGraphContextSteer (id steer_ctx_<chatId>)
- core/graph-context.ts: brute-force cosine top-K, greedy chars/4 fill
under budget, most-recent UnderstandingLevel fold, count cap 64,
parse-don't-cast at the marker seam
- server: embedQuery (role-asymmetric query format, same warm singleton
— vector-space parity by construction); the runner (startGraphContext-
Injection one-shot gates; runGraphContextJob validates ledger parity);
wake branches (chatIdOf/jobIdOf, finalizeJob done, recoverStaleJobs
re-deliver or contextRespawn); abortJob kills context jobs SILENTLY
(nothing was promised in the lesson); turn-service trigger seam
- adapter: data-graph-context flattens to the canonical block (malformed
parts have no model face)
- UI: the budget slider (home dock + /chat/new draft, the session-data
strip); the quiet SystemLine "~537 · 10 propositions"; dock kind
"Knowledge context"; model-picker PUT now spreads settings (a model
pick can no longer clear the sibling override)
Live-smoked end to end on the real embedder + real GLM: seeded ledger
(10 propositions), session on "modal logic: possible worlds and the
necessity box" — job in the dock, queued behind the prober's question,
delivered on the answer, SystemLine in the transcript, wake turn reads
the block and continues the probe. Gates: tsc, lint, 459/459, build.
Louvain only ever found the reasoning chains (the graph is disconnected
by nature — every connected component became its own cluster) and
edge-orphans could never join anything. Clusters are now RELATED TOPICS:
propositions grouped by meaning.
- Embedder (src/server/embeddings.ts): in-process transformers.js v4
running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download),
MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs,
L2-normalized so cosine = dot. Model/dim mismatch voids the ledger
(full re-embed). Injected at the composition root — the agent layer
never imports the server layer (architecture law).
- Ledger (data/graph/embeddings.json): model + dim + vectors + the
persisted cluster partition. Reclustering runs at the END of every
embedding batch (post-merge hook, both front doors) — never per read
(the agglomerative pass is ~1.5s at 338 nodes).
- Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage
agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers
stay singletons by default; attachStragglers is the all-in toggle.
Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans
land in real topics, 10 stragglers.
- Consumers (snapshot, settle, namer, replay) read the persisted
partition — one cluster concept everywhere; core-survival label
matching unchanged (20/26 topics kept labels).
- Louvain retired: detectCommunities + graphology* deps deleted
(graphology-louvain had leaked into the client bundle since the
clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed.
- scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s
including the one-time model download).
The layout now EVOLVES instead of reshuffling on every extraction. A pure
core module (src/core/layout.ts) warm-starts forceAtlas2 from the positions
ledger (data/graph/positions.json, read-modify-write like sources.json,
updatedAt stamp for future live-physics conflict handling), re-blends every
previously-positioned node toward its anchor each iteration (retention 0.85
- old placements stay recognizable, strong forces still win), seeds new
nodes near their positioned neighbors, and seeds orphans on the rim outside
the core so they read as unconnected knowledge.
- Port/store: GraphStore.readPositions/writePositions; snapshot carries
positions when the ledger exists (optional - fresh graphs unchanged).
- Extraction pipeline settles synchronously after every merge (chat and
notes paths), before usage persist - the next /api/graph fetch already
reflects new nodes.
- View: the ledger IS the layout; the local one-shot FA2 pass survives
only as a fallback when no ledger exists.
- scripts/replay-layout.ts: simulates organic growth by replaying the
current 338-node graph one imported file at a time (real import order,
edges activate when both endpoints exist). Real run: 338 positions in
0.3s, 90-degree-0 orphans collecting at mean r=338 vs connected r=194.
- Also restores the cluster region fills (hull background colors) that
6692b19 had removed on a misreading of "no glow".
Live-verified: fit zoom derives from the ledger bbox, pickAt aligns with
ledger positions at every probe, hover/click/detail-card work, 408/408
tests, typecheck/lint/build green.
Future live physics stays cheap by design: the same pure function is the
merge-time settler and the future client simulator; a pinned node is just
an anchor with full retention.
Core-survival labelings (vision section 1, DECIDED 2026-09-13):
ClusterLabeling records in a new append-only ledger
(data/graph/cluster-labels.jsonl), immutable, id = clu_ + sha256 of the
sorted core so re-naming races and re-runs dedupe at the store seam.
matchClusterLabelings (core/clusters.ts, pure): a community keeps a
record's label while at least half its core survives in it; both sides
single-successor (a split keeps one name; greedy by overlap, ties to
the newer record).
The namer (src/agent/subagents/cluster-namer.ts): ONE singular cheap LLM
call (new 'labeler' role, glm-flash/low) names every new/changed
community at once, given member names + statements (capped at 12
members per cluster). refreshClusterLabels runs POST-EXTRACTION
(user-decided) from both front doors, fire-and-forget — naming failure
never fails the run; refreshes serialize behind a chain so parallel
imports never mint duplicate labels. import-notes.ts awaits
clusterNamingSettled before exit.
Payload: GraphSnapshot.clusters (>= MIN_CLUSTER_SIZE 3) joins detection
with labelings at read time — claimed clusters carry the record's label
and id; unlabeled ones carry a member-hash id and null label. Detection
is deterministic so no memo in v1 (ms at current scale; revisit at
vault scale).
Tests (372/372): matcher half-rule/split/merge/greedy/tie cases; naming
end-to-end through importNotes (one call, one record, claim on
re-refresh), naming failure non-fatal; snapshot derivation with
labeling join; 11 test pipelines now pin an explicit labeler seam.
The same extractor→auditor→merger pipeline now runs over raw personal
notes in arbitrary formats (vision §1, DECIDED 2026-09-13):
- NotesSource records (data/graph/sources.json, deterministic id from
path hash → idempotent re-imports) play the chatId role in mappings,
checkpoints, and observation events; usage lands on the source record
(appendSourceUsage, idempotent per startedAt, kept across upserts).
- mergeFindings is chat-store-free: callers pass MergeSource
{messages, foldedAt} — sessions pass the chat record, notes pass
messages:null (no quiz-event rebuild; a deterministic rebuild needs
structured quiz tool parts).
- Second extractor prompt (NOTES_EXTRACTOR_INSTRUCTIONS) for general-
purpose extraction; understanding judges the AUTHOR's grasp. Auditor
wording input-agnostic. extract/audit share one generic runSubmitAgent.
- graphSnapshot.fileSources (srcId→filePath); notes sources render as
static path entries in the detail card (FileTextIcon, never a link).
- One shared pipeline instance (exported from chat-handler): same
single-flight registry, live status chips (label = file path), usage
accounting, and true abort for both front doors.
- Transport: POST /api/graph/import (multipart, 3-wide worker pool) +
Import notes… button in the graph view; scripts/import-notes.ts for
batch runs (concurrency 4).
- real-data-extraction/ gitignored (raw import sources never committed).
355/355 tests; typecheck, lint, build green.
The model-control wave (user-decided design, 2026-09-12): ONE controller
for the conversation agents (prober/teacher are functionally one agent);
background sub-agents stay config-pinned. pi-ai owns the registry — the
shipped 366-model OpenRouter catalog (cost, context, modality,
thinkingLevelMap), getSupportedThinkingLevels/clampThinkingLevel for the
per-model thinking truth, and streamSimple's automatic clamp. No second
registry to drift out of sync.
Seams:
- config.ts: ReasoningEffort retired for pi's ThinkingLevel
(off|minimal|low|medium|high|xhigh|max). GLM roles go honest at "low" —
pi's catalog says GLM-5.3-flash accepts exactly low|high|max, and the old
"minimal" was silently clamped to low by streamSimple since the swap;
nothing changes behaviorally. visualMaker → qwen/qwen3.8-flash (fixes
the dead qwen/qwen3.8-max slug — the catalog only ships -0902; also the
user's cost pick, 0.15/0.47 per-1M).
- src/agent/models.ts: piEngineFor(role, settings?) — provider+id resolved
from the catalog, thinking level from settings ?? config, CLAMPED to the
model's supported list (a settings change between models can never send
an invalid level). PiEngine gains `level` (observability/tests).
- ChatRecord.settings {model?: {provider, id}, thinkingLevel?} — written
by PUT /api/chats/[id]/settings (replace-whole, no updatedAt churn),
read by chat-handler at turn composition: agents are stateless per turn,
so the change is next-turn-effective by construction. The turns log
still records the ACTUAL billed model (the honest witness).
- GET /api/models: the configured-provider catalog as client-safe DTOs
(core/contracts/models.ts) + config defaults; a second provider joins
the picker by registering it in piModels() — nothing else changes.
- PUT settings validation gate (src/server/model-catalog.ts): unknown
provider/model or unsupported level → 400 carrying the supported list,
never a silent clamp.
- scripts/sync-pricing.ts: regenerated PRICES/CONTEXT_LIMITS from pi-ai's
shipped openrouter.json (offline; same source as the engine + picker).
All 366 models priced; glm's stale 0.15/0.5 → pi's 0.075/0.25, limit
1310720 → 1048576 — cost is inferred at read time, so history re-prices.
- UI (components/model-picker.tsx): stats-bar right side becomes model +
thinking buttons; menus open ABOVE; fuzzy search over "provider id
name" (core/fuzzy.ts — pi's /model pattern); model rows show name/id/
price/context; the thinking menu lists exactly the chosen model's
supported levels (GLM: low|high|max — no dead options, no "off").
Model switch drops an incompatible pinned level (falls back to config,
which clamps honestly) rather than carrying a 400.
Tests (350/350, +15): catalog projection + defaults, the settings gate,
pi thinking truth pinned against upgrades (glm low|high|max; minimal→low;
off→low — GLM cannot disable thinking), the engine factory's
override/fallback/clamp/throw matrix, fuzzy ranking, store round-trip,
and the re-priced folds (usage, stats bar, activity dock).
Live smoke (disposable driver, deleted): catalog route 366 + defaults;
glm+minimal → 400 + supported; turn 1 billed qwen/qwen3.8-flash under
the override; mid-session switch flipped turn 2 to glm; clear-overrides
restored the config pin. ALL CHECKS PASS on real OpenRouter.
The custom-message scheme (user-decided 2026-09-12): system-authored
user-role messages are NOT human messages — they carry marker data parts
only, the adapter owns their model-facing wording, and the UI renders
them under control. Nothing fake is ever persisted as human speech.
The inventory (all migrated):
- probe→teacher handoff wake: was a text-bearing user message rendering
as a fake user bubble; now data-probe-handoff marker-only — the
adapter flattens it to HANDOFF_TEXT (core/contracts/chat.ts), the UI
renders a quiet 'Preparing your plan…' system line.
- plan accept/decline: was boilerplate text ('The plan is accepted…',
'I'd like the plan revised…') masquerading as learner speech — the
decline wasn't even marked; now data-verdict {accepted, at} on its own
message, adapter flattens to verdictText(), UI renders the quiet
verdict marker ('✓ Plan accepted' / '↻ Plan revision requested').
sendVerdict sends TWO REAL MESSAGES when the learner typed a note
(the note as a genuine user message, then the verdict) in ONE turn
via setMessages + sendMessage() — no 409 dance, no double teacher turn.
- sub-agent steers: the abort steer gains its data-subagent-aborted
marker part; the UI keys steer rendering off PARTS (artifact/aborted),
not metadata — metadata stays as pump mechanics only.
- record-adapter: systemTextOf() flattens marker-only messages; steers
keep their text (payload — briefs are content, not boilerplate).
phases.ts: plan→teach advances on data-verdict{accepted:true} only;
learner text alone can never sanction teaching (pinned). data-plan-
accepted retired, no back-compat (user-decided: wiped-data doctrine).
Pinned: record-adapter.test.ts (verdict/handoff flattening, note+verdict
order, steer payload kept, UI-only parts skipped), phases.test.ts
(verdict marker semantics, out-of-order, text-never-fakes), jobs.test.ts
(abort marker part), demos.test.tsx (badge renders off the part, not
metadata).
Live GLM smoke (disposable driver, deleted): probe→conclude→wake→plan→
decline-with-note (revised, still plan)→accept-with-note (teach) — ALL
CHECKS PASS, including marker-only handoff/verdict on the wire and zero
fake human speech in the transcript.
Gates: tsc, lint, build, 335/335.
The architectural answer to the validation-crash question: validation IS
first at the model-facing boundary (pi validates arguments before
execute; failures return to the model same-turn — all three live
incidents retried successfully). But the same bytes also exist UNVALIDATED
by design in two other places: the live wire streams raw tool-input
deltas DURING generation, and the transcript keeps the raw input of
REJECTED calls (honesty + the replay invariant). The consumers were
supposed to trust part STATES — but that trust was enforced by
TypeScript casts ('part.input as X'), which are compile-time lies with
no runtime check behind them. That is the hole both crashes came
through, and it should not have been possible.
The structural rule now: model-authored INPUT is consumed only through
parsers running the SAME zod schema the model faced — boundary-valid ⇔
consumer-parseable, mechanically. Server/learner-authored OUTPUT stays
trusted (our code authors it).
- src/core/tool-parts.ts (new): parseQuizInput (display parse over the
shared quizInputBase — strips the teacher-only nodeId, accepts both
prober and teacher boundary shapes), parseAskInput, parseProbeDoneInput.
- contracts/probe.ts: probeDoneInputSchema is now core-owned zod — the
probe_done tool declares it (single source of truth), the seam runs
the literal same schema.
- question-dock.tsx: the dock parses its input at entry; unparsable
renders the quiet malformed-question fallback, never a crash.
- probe-done-card.tsx: the input preview goes through the parse; the
old state reasoning and shape guards collapse into it.
Pinned in tests/tool-parts.test.ts: both exact live crash shapes
(stringified quiz options, stringified assessment), boundary-parity
cases for all three schemas, nodeId stripping, mid-stream partials;
render-cards pins the dock fallback markup.
Gates: tsc, lint, build, 328/328.
The Vercel AI SDK survives only as Band D (useChat client + UIMessage
record types); @openrouter/ai-sdk-provider is uninstalled.
- src/agent/models.ts: PiEngine {model, streamFn} via createModels +
openrouterProvider subpath; thinkingLevelFor maps config effort none->off
- src/agent/lh-tool.ts: neutral LhTool (zod authoring, optional execute,
clientQuestion flag); toPiTool converts via z.toJSONSchema and re-runs
zod inside execute so refinements ride the error-result retry path
- src/server/pi/record-adapter.ts: UIMessage records -> pi Messages
(dangling tool calls get synthesized interrupted results)
- src/server/pi/ui-stream.ts: SSE bridge emitting AI-SDK-shaped chunks so
the useChat client parses unchanged (continuation id rule, question
tools never emit output, tool-input-error for bad authoring)
- teacher-agent.ts: stateless per-turn pi Agents; shouldStopAfterTurn
stops only when every question-tool call resolved cleanly (errors retry)
- chat-handler.ts: buildAgent(outgoing) factory seam (stateless doctrine);
assertValidUiMessages owns the retired SDK validation gate
- sub-agents via shared runSubagent (pi-run.ts); extraction stages stop
after a clean submit; visual-maker keeps its hand-rolled loop over
direct streamSimple calls; aborted model calls bill no usage
- tests: all model seams ported to pi-ai faux provider (scripted
FauxResponseSteps, agent factories); handleChatRequest second param
is now an agent factory
Gates: tsc, lint, build, 305/305. Live GLM smoke (disposable driver,
deleted): prober streaming + persistence, question seam, turn-2
continuation same message id, per-turn real token usage - ALL PASS.
Data wiped at the pivot (chats/graph/jobs, .gitkeeps kept).
The accept-plan UI lands (docs/probe-teach-split.md §4, formerly the
interim rule): presenting the plan no longer moves the phase — the
learner's ACCEPT does. While the presented plan awaits its verdict, the
composer's send square becomes two stacked buttons (green Accept plan /
red Decline); typed text rides the verdict as his note, Enter keeps
sending plain side questions (phase unmoved), decline asks for a
revision and the buttons stay.
The seam: accept sends a user message carrying a data-plan-accepted
part (PlanAcceptedData in contracts/chat.ts, TeacherUIDataParts
'plan-accepted'); phaseAt3 reads it as the plan→teach event (plan_init
demoted to presentation-only); convertToModelMessages drops data parts,
so the teacher sees only the text. Mid-lesson rewrites need no fresh
verdict — acceptance happens once, at the first presentation.
The teacher's planning prompt now ends the turn at presentation and
waits for the verdict (never teaches into an unaccepted plan).
Gates: tsc clean, 304/304 (phases tests rewritten to the acceptance
seam incl. decline/side-question/failed-init cases; composer verdict
render tests), lint, build. Live GLM smoke ALL PASS: phase pinned at
'plan' after plan_init, presentation turn carried NO check, accept
click flipped the summary to 'teach', then tagged checks moved the map
(current=rates, visited=composite); zero console errors.
Diagnosis (2026-09-11 user testing, minimal-effort GLM): sessions stuck on
'plan' forever. Two causes: (1) the three-phase view read the stored tree's
currentNodeId, which nothing ever writes post author-once (2026-09-03) — a
structural dead seam; (2) the teacher stochastically left quiz.nodeId
untagged, and done-by-quiz-result semantics would have falsely completed
nodes anyway.
The redesign (docs/probe-teach-split.md) abolishes derived occupancy in
favor of explicit events:
- Two agents over one transcript: createProberAgent (quiz without a
nodeId field, ask, probe_done) and createTeacherAgent (full toolset,
plan_init onward). The chat-handler construction branch keys on a
probe_done part in the transcript. MODELS.prober added.
- probe_done: server-side seam tool — carries the learner assessment
(strands + free-form edge read), refuses while a question is open or
the probe is already concluded, and queues a handoff steer through the
generalized wake pipeline (steers now dedupe by id, not jobId; job
finalization only for job steers). The teacher auto-wakes and authors
the plan; the learner sees the assessment via the probe-done card.
- phases: probe ends at a concluded probe_done, plan ends at a
successful plan_init — strictly sequential, no skipping, no legacy
compatibility (data wiped at the pivot per doctrine).
- Plan map = position, not completion: derivePlanStatuses reads only
tagged quiz CALLS (state/outcome feed nothing); current = latest
tagged node with consecutive duplicates collapsed, visited = earlier
distinct stays (a revisit stays in both — current wins at render).
PlanTree is structure-only; currentNodeId/doneNodeIds and the dead
mutation helpers are gone. No fallback current.
- Prompts split per agent (prompts/prober/, prompts/teacher/ over the
trimmed shared base); graph-context injection removed entirely
(shelved, docs/probe-teach-split.md §5); knowledge-graph context no
longer reaches either agent.
- Plan blocks are full-only snapshots at plan_init (neighborhood
machinery retired; the persistent DAG window is the future UI wave);
usage capture unchanged.
Gates: tsc clean, 294/294 tests (phases/plan-tree/plan-blocks/question-
tools rewritten to the new semantics; probe-done + handoff-wake tests
added), lint, build. Live GLM smoke PASS: prober concluded with an honest
assessment, teacher auto-woke, authored a 10-node plan with the block
attached, then tagged checks on slope-instinct → secant derived current/
visited exactly as designed; grading honest both ways (driver off-by-one
proved a wrong answer grades false).
Live smoke exposed two bugs in the turn usage log:
- onFinish messages carry no ids (withMessageIds fills copies inside
persistTurn), so fresh turns keyed messageId "" — provenance lost.
- The SDK coalesces continuation turns into ONE response message (a whole
probe conversation lands in a single assistant message), so per-messageId
idempotency swallowed every continuation turn's usage.
TurnUsage gains id: string (unique per request, the append idempotency
key); messageId stays as transcript provenance, taken from the id-filled
list. appendTurnUsage dedupes by id; continuation turns with the same
messageId each keep their usage.
Extraction is real spend and background work; the dock needs to see
and stop it. User decisions: persist all cost data (chat record),
per-stage usage granularity, and TRUE cancellation for abort.
USAGE CAPTURE (live + persisted):
- extraction.ts: per-call onStepFinish captures each billed step's
usage (SDK LanguageModelUsage -> CallUsage: input/output/cache
read+write/reasoning) into the registry's live per-stage usage
(extractor/auditor; the merger is deterministic — no model call).
- /api/graph/status runs now carry usage (model + steps per stage);
cost/context folds are the consumer's (core/prices.ts — nothing
stores dollars).
- ChatRecord.extractions: append-only per-run usage log (mirrors the
turns log): startedAt key, at, durationMs, merge stats when the
merger ran, stages. Persisted on success AND failure — a failed run
still spent money. Idempotent per startedAt; the list payload
(ChatSummary) never leaks it.
- EmissionStats moved to contracts/graph.ts (its contract home);
core/graph.ts re-exports for existing importers.
- Verified end-to-end against the real pipeline (backfill one chat):
per-step token counts, reasoning tokens, and the auditor's growing
prompt-cache reads captured and persisted.
TRUE ABORT (the dock's abort button, user decision):
- RunningExtraction owns one AbortController per run, shared across
phases; abortExtraction(chatId) fires it and returns whether a run
was aborted (idempotent no-op otherwise). The pending model call
receives the signal — the provider cancels the request, spend
stops. throwIfAborted() at phase boundaries ends runs aborted in
sync gaps.
- Abort is a deliberate stop, not a failure: same ledger discipline
(nothing merged, nothing checkpointed, partial usage persisted) but
the run reports zero emissions instead of throwing. Exported via
src/server/observability.ts for master's /api/activity/abort route.
- Tests: mid-run abort (hanging mock that rejects on the signal),
live-accumulation + persisted-record assertions, store-seam
idempotency, and the summary-payload leak guard.
Wave 4 (agent-side observability + control), reconciled with the dock
contract (master relay 2026-09):
- abortJob in src/server/jobs.ts, exported through the server barrel for
POST /api/activity/abort (master's thin route): fires the job's
AbortController (real kill — the in-flight model call aborts, not just
the next loop check), drops any queued result steer, flips the record
to error "Aborted by learner" + activity "aborted" synchronously, and
queues a one-line abort steer (buildAbortSteer) through the same wake
pipeline so the teacher's promise doesn't hang in the lesson. Shares
the result steer's deterministic id space, so abort + racing result
can never both deliver (steerDelivered dedupe).
- Abort registry (src/agent/subagents/aborts.ts) on globalThis, mirroring
the wake pipeline's module-state pattern; runner registers/unregisters
per run, so the route's module instance sees the loop's controller.
- Runner usage capture: LoopContext (abortSignal + onUsage) threads into
all three kind loops; per-step usage folds via accumulateJobUsage into
JobRecord.usage (model from config MODELS, absent-never-zero), flushed
in runJob's finally so an aborted job's money record survives the kill.
- Contracts: JobUsage token fields optional + lastStep (most recent
call's accounting, for the dock's per-blob context line vs
CONTEXT_LIMITS); SubagentResultMeta.aborted; AbortSteerMessage;
buildAbortSteer; SteerMessage union generalized.
- demo-builder/visual-maker accept abortSignal + onUsage (visual folds
generateText usage per iteration).
- stillRunning/finishJob/jobContext guards: an aborted job is never
overwritten by the runner (terminal no matter what the wake does);
failJob and the pump's finalizeJob keep the flip.
- UI: abort steer renders a quiet aborted chip (job-aborted-badge).
- Prompt 20-jobs.md: one line about aborted jobs.
Tests: accumulateJobUsage folds (undefined-preservation, lastStep),
runner usage capture, real-kill abort (sync flip, silent unwind,
abort steer wakes), usage-so-far survives the kill, aborting a
delivering job drops the queued result steer, outcome reasons,
abort-chip render. 330/330, typecheck/lint/build green.
The 2026-09 observability wave's type-level foundation, landed before the
streams so nothing renders against guessed shapes:
- contracts/chat.ts: CallUsage + TurnUsage (per-model-call token accounting,
one entry per teacher/wake turn keyed by messageId) and ChatRecord.turns —
a DELIBERATELY parallel append-only usage log, not per-message metadata:
the client re-POSTs its message list every turn and its copies never
carry server-stamped fields, so anything inside messages gets clobbered
by the next persist (the wave-3 merge-rule lesson).
- contracts/chat.ts: PlanBlockData — the persisted data-plan-block part
(full DAG on plan_init, one-level neighborhood on plan updates, snapshot
semantics so history renders what was true then).
- contracts/jobs.ts: JobUsage + JobRecord.usage — a sub-agent run's
accumulated accounting, written by the runner.
- core/ports.ts + store/chats.ts: ChatStore.appendTurnUsage — idempotent
per messageId (abort replays / wake retries append once); listChats no
longer leaks turns into the list payload.
- core/prices.ts: PRICES + CONTEXT_LIMITS (generated from OpenRouter's
published pricing via scripts/sync-pricing.ts) + pure folds: callCost,
turnCost, sessionCost, contextWindow, jobCost. Unknown model or absent
fields render as absent, never zero.
- server/observability.ts: barrel-approved surface for the transport layer
(fixes the import-law violation in the graph status route that slipped
through the wave-3 merge verification).
Capture wiring (agent onStepFinish -> persist seam, plan-block emission)
lands next as master seam work; the streams start against these shapes.
Wave 3: the plan protocol collapses to one re-callable mermaid plan_init
(seven mutation tools deleted by design; statuses derive from quiz checks;
live quiz-nodeId validation with mid-turn retry). The steer-starvation arc:
steers deliver coalesced with the learner's answer (claimSteers), persist
dedupes same-id continuation copies, jobs settle-sync defers while a turn
is busy. Research steers gain the collapsed full-brief affordance. Model
reasoning effort set to minimal across roles.
The bug (seen live): a teacher that ends every turn with a new pending quiz
keeps the wake pipeline's idle gates permanently closed — the steer sits in
'delivering result' forever while the learner answers check after check.
- claimSteers (server/jobs.ts): the chat-handler claims every queued,
undelivered steer the instant a client turn wins the turn registry, and
appends it after the learner's answer — conversion-safe, the answered
quiz carries its output. Every answer flushes pending steers immediately.
- The pump keeps its post-idle duplicate-guard (re-check steerDelivered
before firing), so a claim racing the pump converges to exactly one
delivery either way.
- The stale-job sweep now skips chats with a turn in flight (a healthy but
slow wake no longer gets falsely re-queued as 'recovering').
- Out-of-lane surgical touch (flagged for master review): the 3-line
handleChatRequest seam (claim after beginTurn, persist+stream 'outgoing').
- Verified live end-to-end on the stuck session: answering the pending quiz
fired the steer + wake turn within a second; job flipped done/delivered.
- plan_init is now the ONLY plan tool: a mermaid 'graph TD' block parsed by a
constrained-subset parser (mermaid-plan.ts) with line-precise errors; the
seven mutation/tracking tools are gone by design (2026-09-03 design pass).
- Re-call plan_init = sanctioned wholesale rewrite; slugs are permanent.
- derivePlanStatuses (contracts/plan.ts): done = passed check tagged with a
quiz nodeId; current = open check's node else first topo-ready node — the
tracking the teacher no longer declares by hand.
- questionToolsFor(ctx) (tools.ts): the quiz schema chains a dynamic plan-
nodeId validation against the LIVE record — an invalid nodeId is rejected
at the call boundary and the teacher retries mid-turn, before any UI.
Static questionTools stay validator-less for the default path.
- Prompts rewritten for mermaid authoring + derived position.
- Out-of-lane surgical touches (flagged for master review):
src/core/question-core.ts (quizInputObject export + nodeId description),
src/core/contracts/plan.ts (derivePlanStatuses + doc).
The extractor's per-session understanding notes were free-form strings;
the model freely appended em-dash justifications ('shaky — the learner
could state the headline distinction but calls the machinery behind it
foggy'), which overflowed the detail card's session rows and squeezed
the session names.
The note becomes a CLOSED named-word scale — solid / shaky / accepted /
resisted / not yet engaged (DECIDED 2026-09-08, no numbers) — enforced
by the zod enum in propositionSchema: a wrong-shaped note is rejected
at parse time and the agent retries. Justifications are dropped, not
stored (user decision: not necessary). Extractor and auditor
instructions updated to demand exactly one word.
Renderer: the session row keeps the session NAME as the main thing
(flex-1, truncates) and the level chip is bounded (max-w-28,
whitespace-nowrap, truncate) — overflow impossible even for legacy
prose notes in old ledgers.
Vision doc: §1 extractor bullet + event-ledger line updated; state
model line notes the five-class derivation was removed; amendments
table row added.
The extractor's per-proposition understanding observations (named-word
scale: solid/shaky/accepted/...) were ledger data since the redesign,
with no surface. This exposes them at read time and shows each
session's note as a small chip on its session row in the node detail
card — the per-session confidence history the user asked for, one
observation per (node, chat), latest wins within a chat.
- GraphSnapshot.nodeObservations: nodeId -> chatId -> note, built in
graphSnapshot() from observation events (pure read-time join).
- Sessions list rows: link + observation chip.
- Session-relative judgments stay session-scoped: they render next to
their session, never as node state (the state model stays pending).
The five-class NodeStateClass (taught/known/missed/dont-know/candidate)
was transitional scaffelage from the 2026-09-02 redesign, deriving state
by latest-quiz-wins — exactly what the philosophy rejects: understanding
is inferred at read time from the real state model (still pending
ideation), never encoded as quiz-derived classes.
- contracts: NodeStateClass deleted; GraphSnapshot no longer carries
nodeStates (objective graph + sessions only).
- core: deriveNodeStates replaced by latestQuizOutcomes — the raw
event fact (nodeId -> latest quiz kind), consumed ONLY by the
teacher's graph-as-context selection, never by the home renderer.
- graph-context: hotspot selection re-keyed to outcomes — groups are
now 'Hotspots (last answer wrong or don-t-know) / Answered correctly /
Never probed'. Same diagnostic selection, no state taxonomy.
- graph-view: nodes render uniformly (accent color, size 8); the panel
is search-only; the detail card's state badge is gone. Quiz outcomes
remain ledger data for the future state model.
Verified live (headless): uniform discs (single color/size, zero state
pixels), search-only panel, card = name + statement + sessions, no
console errors.
With real extracted data, full-sentence statements as canvas labels proved
unworkable: 10-line label blocks, illegible overprint clusters, bottom-edge
clipping. Per the user's 2026-09-08 decision:
- GraphNode/ExtractedProposition gain `name` — a very short, glanceable
label (glossary-headword style: "Will to power"), presentational only:
not unique, not identity, never precise. Set at creation; link-appends
never change it (first line defines; labels stay stable across sessions).
- `statement` keeps its full precision and becomes the identity anchor;
the detail card shows it under the name. `description` is dropped.
- Extractor coins the name (freshest session context); auditor carries it
through on create, rewording only clearly unglanceable names; linked
propositions keep the existing node's name.
- Renderer: canvas label = name; search matches name + statement;
detail card = name (bold) + statement (muted).
- Store guard requires name (pre-pivot node lines are skipped as legacy);
vision.md node-schema bullet + amendments table updated (2026-09-08).
The minimal implementation of the 2026-09-02 graph vision (docs/vision.md §1):
- contracts/graph.ts: every node is a TRUE proposition (statement is both
rendering label and identity key; required description; no kind, no name).
Edges are follows (derivable) / supports (grounds without entailing) with
message-granular provenance points {chatId, messageId}. Events reduce to
correct/miss/dont-know plus observation (understanding notes); no taught
events, no misconception kind.
- extraction.ts: extractor is TREE-BLIND (transcript only, no plan tree) and
free-form; auditor sees findings + the current graph, owns faithfulness and
identity (link-or-create verdict per proposition, recorded as linkTo);
transcriptView tags every line with its message id (provenance anchors).
- core/graph.ts merger: applies verdicts deterministically — linkTo appends
a link line (phrasings join aliases, provenance unions; reads merge by id),
absent creates; NO string matching anywhere. Quiz events rebuilt from tool
parts with attribution-only nodeId resolution; understanding notes become
observation events (obs:<sourceId>); mapping lines are the verdict audit
trail and make re-merges idempotent.
- deriveNodeStates: transitional — latest quiz event wins (known/missed/
dont-know, else candidate); taught unreachable until the state-model
ideation lands.
- graph-context.ts + 50-graph-context.md: follows chains, hotspot grouping,
state vocabulary updated for the new event kinds.
- graph-view.tsx: node label is the statement; detail shows statement +
description.
- tests: merger verdicts (link, no-string-match, cross-chat assertions),
tree-blind pipeline end-to-end, follows vocabulary in context tests.
276/276 green.
Purely mechanical: src/core/contracts/{plan,graph,chat,jobs}.ts carry the
sections verbatim (graph = nodes + ledgers + extraction findings; jobs =
the async ack-and-steer protocol + tool results + demo bridge), and
contracts.ts becomes a re-exporting barrel so every existing import path
(@/src/core and ./contracts.ts) is unchanged.
Wave-2 lane ownership follows the split: the graph stream owns
contracts/graph.ts, the teacher stream owns contracts/jobs.ts — same-file
contract collisions between streams are structurally impossible now;
cross-domain schema changes go through master.
amosblomqvist