Owner ruling 2026-10-04: the module is a sunk cost, retrievable from
history if ever revisited. Removed: core practice contracts + pipeline
helpers, the PracticeStore port + adapter + data dir path, server
service + API routes (sets, clusters picker, practice aborts), the
practice pages + dialog/exam/list UI, chain-table's Generate handoff,
sidebar nav, activity-dock practice rows, and the partitioner/
problemGenerator/grader model roles (BACKGROUND_MODEL_ROLES 8 → 5).
Survivors, moved to core/chains.ts (pure + browser-safe): ReasoningChain,
chainsForSelection, formatChainLine — the chain table and graph
emission still consume them.
pi-ai 1.0's onProviderStreamEvent, at the layer our incident class lives
(GLM arg truncation, DeepSeek id resets) — env-gated (owner choice:
no app surface, off by default; raw events carry full prompts), one
JSONL per chat under data/debug/provider-events/ with a 20MB cap and a
run-start marker for record correlation.
Wired at every agent construction: teacher/prober turns (AgentBuildOptions.
chatId), jobs via LoopContext (researcher, demo-builder through
runSubagent; visual-maker + pdf-reader through their streamOnce), and
the extraction pipeline (extractor + auditor keyed by source id).
practice deliberately unwired (sunk-cost module).
Owner live-test verdict on the spine (2026-09-30): no perceptible gain —
the real problems (restatement nodes, shallow re-derivations, same
conclusion from many angles) are mint-time issues a post-hoc chain
minter cannot fix. This arc moves assembly INTO the extractor and adds
the deterministic dedupe the owner specified:
- Digest-aware extraction: each stretch's extractor prompt carries THE
GRAPH SO FAR (its own source's merged propositions + chains, ids
verbatim); build on existing ids instead of restating, lean on
established conclusions as premises. Mixed-namespace chain endpoints
(fresh p-ids OR existing n_-ids) validated at the submit seam.
- Serialized per source: a source's batches/chunks run strictly in
order (each extract reads every earlier merge); cross-source runs
stay parallel; source-wide abort stops the loop. The spine stage,
role, phase, usage stage, cost row, and setup roster are gone.
- Core-ideas prompt: goal reframed as uncovering core ideas and showing
how everything derives from the foundations; direction upward, prefer
lean derivations, never re-derive what the graph already holds.
- Chain simplification (merger maintenance): simplifyChains drops
provably redundant chains at every merge — prefer-fewest-premises +
ancestry-cover ([A,B]⟹X with [X,C]⟹Y covers [A,B,C]⟹Y, transitive);
same-key provenance lines drop together; rewriteLedgers under the
audit lock; droppedChains counted separately in stats.
Chunk budget stays 8192 (owner: keep for now, maybe lower later).
Per-batch pipeline is now extract → spine → audit → merge (owner order,
2026-09-30), the spine inside the serialized audit section so it reads
the graph exactly as the following audit will. Measured motive: 0 of 133
chains in the live corpus crossed a stretch boundary — no stage minted
cross-stretch chains.
- src/agent/subagents/spine.ts: SESSION/NOTES spine preambles + shared
SPINE_CORE (span doctrine; extraction doctrines carried over), the
submit_spine TypeBox schema, validateSpine (mechanical span law: every
chain touches >=1 fresh endpoint AND >=1 existing digest endpoint;
unknown ids, self-reference, and id collisions are loud), the digest
prompt builder, and the runner (strong tier, retry-only).
- extraction.ts: 'assembling' phase, spine stage in the per-need gate,
usage/billing stage, assemble() inside withAuditLock for both front
doors; first stretch of a source skips the call (empty digest, no
spend); the audit sees COMBINED propositions (chunk + connectives),
never chains; merger receives the combined findings.
- core/graph.ts mergeFindings: chain endpoints resolve through the
job-local map OR pass through verbatim as existing n_ ids (the spine's
mixed namespace).
- EXTRACTION_CORE: steps-of-reasoning selection bar (examples, notation
glosses, and restatements that no argument needs are not steps).
- Factory chunk budget 4096 -> 8192 (fewer seams, more local context).
- healStringifiedArrays moved to the shared submit seam (submit-agent.ts).
- Cost panel: Assembly row; usage records carry a spine stage (optional
on legacy records).
- Tests +5: prompt pins, span-law matrix, digest rendering, seeded
end-to-end flow (spanning chain + connective, verbatim n_ endpoint,
spine billed as its own stage), non-spanning refused + retried.
Data analysis of the five-source corpus (490 nodes, 244 chains) found
three pathologies the goal-first core let through:
1. Narrative-adjacency chains — premises that occasioned rather than
entailed conclusions (concentrated in the polemical sources).
2. Session-scaffolding minted as knowledge — the source's own axiom
numbering (A1/F2/UT2), arc labels ('the crux') persisted as nodes.
3. Full-discourse volume — glossary and argument-arc material at equal
rank with load-bearing ideas (~200 nodes per source).
Three doctrine lines, same terse register as the core:
- entailment is the only link that earns a chain; historical/narrative
connections keep their pieces as bare propositions or drop them
- load-bearing ideas only (what a reference on the subject would teach)
- propositions must stand alone for a reader who never saw the source;
no leaning on the source's own scaffolding
Substring pins for all three; shared by session + notes extractors by
construction.
- architecture.md section 6 rewritten: stored truth = propositions + chains
(chains.jsonl); deriveView as the lossy image (implication nodes +
untyped arrows, rebuilt per read); the bijection framing (independent
arguments for one conclusion stay distinct — the corruption the edge
store invited); new deletion-semantics paragraph (proposition death
cascades to every referencing chain; arg_ death = one argument);
supports retirement stated plainly; auditor's propositions-only scope;
clusters stated as birth-session partition truth (no Louvain claim)
- section 7 practice chain model: stored-chains inclusion law replaces
follows-edge grouping; chainsToGraph bijection claim deleted
- README: graph clauses (stored chains, derived view, chain dedupe),
practice clause, ledger inventory, and two long-stale line-5 claims
(ledger events, semantic top-K) corrected
- contracts/graph.ts GraphSnapshot comment: Louvain claim replaced with
birth-session truth; extraction.ts comments no longer narrate the
dead expandFindings/chainsToEdges seam
Owner rulings 2026-09-30: reasoning chains are the stored truth; the graph
canvas renders a lossy derived image. No migration, no backcompat — old
graph data was wiped.
- contracts: GraphChain (arg_-minted id, premises, conclusion) replaces
typed edges; GraphEdge survives only as the untyped derived arrow;
GraphSnapshot carries stored chains; supports edges retired entirely
- store: chains.jsonl ledger (edges.jsonl purged at load alongside legacy
events.jsonl); readChains/appendChain seam; merge dedupes by
(sorted premises, conclusion), re-assertions append under the same arg_ id
- core: deriveView mints implication nodes (name ⟹, statement via
formatChainLine) plus premise→imp→conclusion arrows; clusters stay
proposition-only (birthSessionPartition); chainsForSelection now groups
STORED chains (chain rides iff every member selected) — independent
arguments for one conclusion never union; chainsToGraph/chainsToEdges die
- extraction: extractor submits chains natively (expandFindings deleted);
auditor receives propositions only; supports fan-out gone
- deletion: full death of a proposition kills every referencing chain;
arg_-prefixed node deletes kill that chain; chainsRemoved stats
- UI: every arrow renders the ⟹ double shaft; supports toggle + thickness
slider and detail-card edge sections deleted (card keeps name +
proposition); chain table + practice pipeline read stored chains
- tests: 1188 green across the new seams
Rulings: each problem generation is its own visible dock job (the
extraction per-chunk pattern), and practice shares extraction's
semaphore — one process-wide 3-permit pool for ALL background LLM
jobs; total background LLM concurrency bounded at 3.
A. SHARED POOL: new src/agent/subagents/bg-llm-semaphore.ts —
globalThis __lhLlmSlots, PERMIT_COUNT 3 (the ruling), the
fresh-take/handed-off-permit deadlock discipline preserved for
BOTH callers. extraction.ts's withExtractSlot is now a thin
wrapper (job-row signal → withBgLlmSlot); __lhExtractSlots and
__lhPracticeSlots both die. The partitioner now acquires a permit
too (it's a background LLM job — it didn't before).
B. REGISTRY ROWS: the aggregate setId:generation / setId:grading rows
are replaced by setId:partition (phase partitioning, label
'Partitioning — title'), setId:gen:problemId (queued→generating,
'Problem i/N — title'), setId:grade:problemId (queued→grading,
'Grading i/N — title'). Each row carries its OWN abort controller
and its OWN usage (one PracticeStepLog per row — the aggregate
partitioner/generator/grader folding dies). Row key = feed id =
abort key (the dock dup-key law).
C. ABORT SEMANTICS: abortPracticeRow(key) routes by shape — gen row:
THAT problem only (error 'Aborted', siblings continue, set settles
ready with one errored problem); partition row: the whole run;
grade row: that grade drops (answer kept, re-submittable).
abortPractice(setId) composes from row aborts + the synchronous
persisted marks (a set-abort's verdict stands before the settles
re-read it — the settle step keeps an existing error status).
/api/activity/abort kind practice now takes the ROW key (charset
admits ':').
D. FEED + DOCK: ActivityPractice/FeedPractice carry per-row identity
(id = row key, problemIndex/problemTotal); dock blobs render per
row, labels carry the i/N; the aggregate row is gone.
E. Boot sweep unchanged at set grain; the live-row check is now a
key-prefix scan (any live row of the set protects it). The
updatedAt heartbeat covers the new row scheme as-is.
Tests (20 in the pipeline suite): per-problem row shape/uniqueness/
i-N labels (partition row frozen behind a gated partitioner; per-row
usage ISOLATION pin — b's log never shows on c's row); abort-one-gen
(sibling completes, set ready with one errored problem); partition-
row abort; queued grade-row abort (dead-ticket path: answer kept,
re-submittable, sibling lands); shared-pool identity + contention
(practice queues behind extraction-style holders); registry/dock
feed mapping. Test isolation hardened: unique per-store set ids,
afterEach registry/lock/pool reset (a timed-out test can no longer
poison the rest).
architecture.md: jobs-pattern paragraph rewritten (per-problem rows,
shared pool, abort routing).
1149/1149, tsc 0, lint 0, build 0.
runSubmitAgent moved verbatim from extraction.ts to src/agent/subagents/
submit-agent.ts (exported; extraction imports it — zero behavior change,
extraction suites green untouched). New optional prepareArguments seam:
extraction wires its healStringifiedArrays heal; default is identity
pass-through. ModelRole += partitioner | problemGenerator | grader;
ROLE_SLOT pins all three to strong (quality-critical trio, plan default).
Note: BACKGROUND_MODEL_ROLES (per-role settings overrides) deliberately
NOT extended — settings-surface decision, flagged for the planning
session.
Observed live: the auditor looped find for a very long time, never touching
list_sessions/list_propositions. Root cause was the prompt: its only
procedure text was 'For each proposition, search the graph (its key words,
its synonyms, its phrasings)' — a per-proposition find loop with no
starting point, no sense of the space's extent, and no termination rule.
And find returned id+name+statement with no session pointer, so results
never composed with the listing tools (owner diagnosis, 2026-09-29).
- AUDITOR_INSTRUCTIONS now prescribes the process: (1) list_sessions ONCE —
the graph is small and finite, this shows its entire extent; (2) read
plausibly-related sessions with list_propositions (full text) — every
realistic candidate seen in a few calls; (3) find as the targeted sweep;
(4) SUBMIT once related sessions are read — never re-search seen ground
- find hits now carry 'born in: <session id>' — results point at sessions
the auditor can read whole (legacy no-provenance nodes name none)
- identity doctrine untouched: equivalence bias, linkTo mechanics,
verdicts-only deliverable
Diagnosis on tonight's data (4 sources, 173 nodes, 216 edges): supports is
62-83% of edges, topology is hub-and-spoke (34/94 libertarianism nodes are
degree-1 leaves), hub edges read as essay-flow not logic, 33-41% of nodes
are degree<=1 across sources. The old doctrines asked for exactly this:
'every distinct proposition' + 'lonely nodes = failed extraction' +
'COMPLETENESS... omitting is the main way extraction fails'.
Owner rulings, prompt-only (data model untouched, both prompts):
- strict reasoning bar: a proposition earns its node by being a premise,
step, or conclusion of an argument the source actually makes; isolated
facts, name-drops, asides, trivia, example enumerations are dropped
- supports tightened: ONLY explicit grounding acts in the text, rare
exception, genuinely argumentative relations are almost always follows
- completeness inverted: precision over recall, WHEN IN DOUBT OMIT,
never connect merely-related propositions
- MUST-connect clause dropped; corrections of misconceptions stay in
(argumentative by nature)
Identity doctrines untouched (established-terminology names, MERGE YOUR
OWN OUTPUT, aliases) — zero near-duplicate statements proves they work.
The persisted (chatId, sourceId)→node table was a shared namespace across
independent chunk/batch jobs: each chunk's extractor mints local ids fresh
(p1..pN), so later chunks collided with earlier ones — propositions silently
skipped as 'already resolved' (coverage checkpointed anyway: permanent silent
loss) and edges resolved onto the wrong nodes (the 'everything links to
everything' mush). Owner ruling 2026-09-29: mergeFindings resolves THIS job's
ids in memory, nothing persists between jobs; cross-job identity is the
auditor's linkTo verdicts alone.
- contracts: NodeMapping dies; NotesSource.contentFp — sha256 of the text at
the last run, the re-import guard (unchanged re-import = TRUE no-op, no
model calls, no usage; changed text re-runs; legacy unstamped sources
re-run once)
- merger: job-local resolution map, in-submit duplicate ids skip once,
edges resolve extractor-local ids only
- ports/store: appendMapping/readMappings, LEDGER_FILES.mappings,
rewriteLedgers.mappings gone (stale data/graph/mappings.jsonl just stops
being read)
- session-clusters: birthSessionOf is provenance-only — the legacy
mapping-line fallback for provenance-empty notes nodes dies
- deletion: no mapping rows to strip; stats lose mappingsRemoved
- replay-layout batches nodes by first provenance point
- graph.test pins the new independence semantics (re-merge re-creates;
duplicate ids within one submit resolve once); extraction tests pin
per-chunk merges + the fingerprint guard
Hypothesis under test by the owner: coherent extractions turn to mush
after merging because the auditor minted too many bridging edges of its
own — inference, not identity, and out of the auditor's scope. The graph
now connects across sources solely through identified equivalences;
relations between propositions come from the extractor alone.
- auditSchema drops 'edges' (submit_review = links only); the submit-seam
required-field doctrine reworded per tool
- AUDITOR_INSTRUCTIONS: the BRIDGING EDGES section dies; deliverable is
the identity verdicts only
- AuditVerdicts drops edges; ExtractedEdge endpoints are extractor-local
ids only; mergeFindings stops consuming verdict edges and its resolve()
drops the raw-node-id branch (dead — the extractor never sees graph ids)
- extractor untouched: submit_findings keeps propositions + follows/
supports edges, required as before
- tests: 17 submit fixtures + 9 typed verdict literals slimmed to links
- docs: architecture.md division-of-trust + pipeline diagram updated
Every notes chunk and chat coverage batch is its own job — registry row,
usage record, failure domain. Extracts fan out process-wide behind a
3-permit extract semaphore on globalThis (new 'queued' phase); audits and
merges stay serialized behind the existing FIFO gate ('waiting'), so each
auditor explores the graph after every earlier job's merge.
- ExtractionPhase gains 'queued'; RunningExtraction gains part {i,n};
the feed shows one row per job with a '· i/N' suffix (status route)
- abortExtraction is source-wide: every job of a fan-out dies together;
merged jobs keep their coverage checkpoints
- failure isolation at the fan-out (Promise.allSettled + aggregation):
a failed chunk no longer kills its siblings — the run returns what
actually merged; job-level aborts log, pre-fan-out errors still throw
- usage ledgers: one record per job (source-keyed), failed jobs bill
their partial spend
- runJob/aggregateJobs helpers; setPhase/recordStep/extract/audit/
notesExtract key the registry by jobKey (src#i) — notesExtract's
signal lookup was the one real bug the new tests caught (source-id
lookup meant fan-out aborts never reached the model call)
- semaphore is handoff-correct (fresh takes increment; handed-off
permits don't double-count — an inflated counter deadlocks at 3)
- tests: isolation pins rewritten (stage failure now resolves zeros),
single-flight/registry tails updated, per-job usage counts, and a new
describe pinning the fan-out: 3 extracting + 5 queued, parts 1..8,
source-wide abort, per-job usage on failure, extract peak == 3,
audit peak == 1
- docs/architecture.md: serialized-audit paragraph rewritten for
per-chunk jobs; two-front-doors wording aligned
Owner ruling 2026-09-28: the auditor has no control over the
extractor's output and never alters it, so re-emitting the full
findings per submit was pure payload cost — the retry collapse the
schema slim attacked, halved again. submit_review is now links
(local proposition id -> existing graph node; EQUIVALENCE BIAS
carried in the linkTo description) + bridging edges ONLY; the
merger joins extractor findings + verdicts.
mergeFindings(chatId, findings, verdicts, ...) — propositions come
from findings verbatim, links join by id, edges emit extractor's
then auditor's through one dedup-preserving emitter. The old echo
form hid an accidental drop channel: an auditor that omitted a
proposition from its echo silently dropped it from the merge.
Verdicts-only removes it structurally — extractor output always
fully merges (tests updated to full-merge expectations).
AuditResult/AuditedProposition -> IdentityLink/AuditVerdicts;
audit() logs links/bridges; batch view renders anchorless like
transcriptView; auditor prompt rewritten to the verdicts-only
deliverable; architecture.md division-of-trust updated.
OpenRouter log diagnosis (owner, 2026-09-28): extraction time was
dominated by tool-validation retries. The failing call had the whole
propositions array emitted as ONE JSON-STRING ('propositions.0: must
be object'), a truncated placeholder artifact inside it, and
word-salad statements late in the array — payload-size collapse.
Field audit: 'evidence' was never persisted anywhere and its
faithfulness-checking consumer died with the fully-trusted ruling;
'messages' fed messageId granularity that NOTHING downstream reads,
and the model sent [] anyway (chat-level provenance in practice).
'aliases' stays (optional now): the auditor's find tool globs them.
Schema: propositions = {id, name, statement, aliases?}; edges =
{from, to, type}. Prompts lose the evidence/messages doctrine and
[id]-anchor guidance, gain a compact faithfulness line;
transcriptView renders anchorless. The auditor echo instruction
drops the evidence/messages mention, so its payload halves too.
Heal: healStringifiedArrays on the submit tool's prepareArguments
(pi-agent-core's sanctioned pre-TypeBox shim) parses a stringified
propositions/edges back into arrays — the LaTeX-restore doctrine:
heal weak-model wire behavior at the seam. Test drives the real
agent-loop seam with a stringified submit.
Extraction may run in parallel across sources (extractors are
source-local and graph-blind), but the auditor explores the LIVE
graph through its tools and stamps identity verdicts against what
it finds — two concurrent audits would each link into a graph the
other is about to change, the structural mint of duplicate nodes.
withAuditLock: a FIFO promise-chain mutex on globalThis (the same
module-instance idiom as the run registries — a module-level lock
would be per-route-instance and useless in dev). The check runs
before any LLM is invoked or the graph read; the lock spans audit
through merge inclusive (the merge is the graph mutation) and is
reacquired per batch, so interleaved multi-batch runs each audit
against the latest graph instead of starving each other. Applied at
both front doors — chat batches and notes imports.
New 'waiting' phase (extracting → waiting → auditing → merging):
visible in the status feed so a queued run never looks stalled in
extracting. Abort-aware on both sides of the queue: a run aborted
while waiting leaves without auditing (its chain link still
releases — successors never hang); a run aborted mid-audit releases
in finally.
Tests pin: a second run queues at the gate and its auditor is
invoked only after the first run's merge is in the ledgers (the
post-merge-graph invariant itself); a run aborted while waiting
never audits, checkpoints nothing, and leaves a healthy chain for a
third run. Gates: typecheck, 1061/1061, lint, build.
Owner's first live test of the relinked pipeline surfaced three issues,
diagnosed against the ledgers:
1. Detail-card edge labels were backwards (UI only — data and canvas
arrows were correct). Semantics pinned 2026-09-28: an edge A→B
means A implies B (follows) or A supports B (supports). Outgoing
edges now list under 'What follows from this' / 'What this
supports', incoming under 'Follows from' / 'Supported by'.
Verified against the ledger: Invariance --supports--> Twin
paradox / Relativity of simultaneity; Argumentation ethics
--follows--> Homesteading — arrows correct, only the words were
swapped. docs/architecture.md wording fixed to match.
2. Session-local terminology leaked into node names ('Criterion C',
'Action axiom (F1)'). Both extractor prompts gain a naming
doctrine: names use established, discipline-canonical terminology
(glossary-headword test); never source-local bookkeeping (letters,
numbering, positional labels); local shorthand goes to aliases.
3. Near-duplicate propositions and lonely nodes (~40% of fresh-run
nodes edge-less; 'Argumentation ethics' vs 'A priori of
argumentation' duplicated despite the auditor having linked that
phrase once). Root cause is upstream of the auditor: the extractor
had no self-merge or edge-completeness duty. Both prompts gain:
MERGE YOUR OWN OUTPUT (same claim at different precision = one
proposition, phrasings to aliases; nothing downstream dedupes) and
edge COMPLETENESS (a proposition that arose as evidence for,
consequence of, or prerequisite of another must carry the edge —
a graph of lonely nodes is a failed extraction).
Owner rulings on the open question: keep two edge types (no
'relates to' junk drawer); re-measure loneliness after these fixes.
Gates: typecheck, 1059/1059, lint, build.
The graph becomes raw reasoning chains only. Quiz outcomes and
understanding observations are no longer formalized; assessment
moves to the future practice module.
Events ledger removed end-to-end: LedgerEvent/LedgerEventKind/
UnderstandingLevel/ExtractionAttribution/AuditResult.dropped and
snapshot nodeObservations deleted from contracts; quizEvent builder
and the events step of mergeFindings gone; appendEvent/readEvents
dropped from the port and store; LEDGER_FILES loses events.jsonl
with a one-time purge of a stale file next to the nodes ledger;
graph-deletion and the delete dialog lose event purge + quiz-answer
impact counts; GraphSessionProposition becomes a pure birth list.
Whole-session markdown source view: new core/session-markdown.ts is
the shared formatting engine (id-anchored speaker blocks, quiz/
recall/question blocks, tool one-liners) — the extractor's prompt
view and the future user-facing export render from one engine.
Coverage fingerprints deliberately stay on the old line format
(content-hash semantics; re-formatting never re-opens covered
history). The whole session goes to the extractor — no phase cut.
Auditor refocused as the identity linker: never sees the source;
explores the graph via new auditor-specific tools in
agent/tools/graph.ts (find — glob over name/statement/aliases;
list_sessions; list_propositions with full statement text; no
timestamps) instead of an injected full-graph dump; equivalence
bias — paraphrases merge, only genuinely distinct propositions stay
apart; no drop channel, the extractor is fully trusted; may draw
bridging edges (typed follows/supports) between existing nodes and
the batch, resolved by the merger with chat-scoped message-less
provenance. Agentic loop, maxTurns 16.
UI: the node detail card gains an edges section above the sessions
list (follows-from / supported-by / what-follows / what-it-supports,
rendered only when present, click-to-select neighbors); the
understanding-note chips are gone.
Prober list_propositions reworded to what a session covered —
names only, no levels. docs/architecture.md + README updated to
the as-built pipeline. Gates: typecheck, 1059/1059, lint, build.
Server: ui-stream.ts is verbatim pi AgentEvent passthrough (SSE data
lines + [DONE]); only app-level events are the loop-level error and
turn-settled terminal; onError carries raw provider text (owner ruling).
Client: useChat + the AI-SDK transport replaced by components/session-core.ts
(SessionModel: 409 identity-retry, optimistic c_ appends, absorbed()
adoption, needsContinuation, shared consume for POST + reattach) and the
session.ts hook; the golden-master grouping canon moved client-side into
core/view.ts (projectView); scanners read the native record (phaseAt3Lh,
danglingQuestionOf, graph toolParts, plan-blocks, runner, extraction).
Coverage: the native message is the unit (fork 3) with the call+result
inseparability bond rule; view-line-less messages (markers, error
results) are invisible to coverage and never checkpointed.
Deletes: chat-transport, projection, ui-message, their tests (ported to
tests/session-core.test.ts), and the ai + @ai-sdk/react dependencies —
the Vercel SDK is fully out of the tree. Wire pins in chat-flow rewritten
onto the AgentEvent dialect (eventsOf parser).
Gates: tsc clean, 994/994 tests, eslint clean, build green; live smoke
on PORT=3004 (verbatim wire → native record).
The chat record is now pi-native agent history (LhMessage[]), the AI SDK's
UIMessage transcript contract is gone from the server, and every client turn
arrives as ONE explicit input event.
Core (new):
- src/core/native-messages.ts — the LhMessage taxonomy (user/assistant/
toolResult + lhMarker customs: verdict, probe-handoff, subagent-steer,
subagent-abort, plan-block), constructors, and toLlmMessages — the single
model-facing seam (convertToLlm): strips ids, flattens markers to user
text, drops UI-only plan-blocks, synthesizes INTERRUPTED results for
dangling aborted calls, never throws.
- src/core/projection.ts — projectNative: the one-way native → UIMessage
projection (consecutive assistant+toolResult runs group into one
assistant view message; toolResults attach backward by toolCallId;
markers project to data parts; plan-blocks splice after the last
successful plan_init). Every existing consumer reads through it.
- src/core/input-events.ts — the ChatInput union (text/answer/verdict/
continue), parseChatInput validation, and inputAppends (answer targets
the dangling question call or 400s; verdict = optional note user
message + native verdict marker).
Server:
- chat-handler.ts — the turn root takes {chatId, input}; write-ahead
persists input appends + claimed steers before the turn; claimSteers
coalesces the whole queue with the handler's steerDelivered filter as
the dedupe seam; teacher-vs-prober branch on hasProbeDone.
- turn-service.ts — persistTurn stamps ids, drops clean question
placeholder toolResults (the record ends at the unanswered call),
emits plan-block markers idempotently (plan-blocks.ts rewrite), and
persists abort partials.
- ui-stream.ts — PiTurnSpec takes the native base; the wire dialect is
unchanged. jobs.ts steers are native markers (steer_<jobId> dedup ids);
waitForIdle reads danglingQuestionOf. teacher-agent.ts wires
convertToLlm: toLlmMessages. record-adapter.ts DELETED wholesale.
- GET /api/chats/[id] serves messages: projectNative(record.messages).
Client:
- chat-transport.ts — the 409 retry is IDENTITY (re-POST the same event
body after the turn settles — the event was never persisted
pre-acceptance; the merge logic is gone). The transport consumes the
stashed input event via prepareSendMessagesRequest and throws loudly on
a missing stash.
- chat.tsx — every send call site stashes exactly one ChatInput event
(text, answer, verdict, continue) before invoking the SDK send.
Tests: full native migration (chat-flow rewritten to input events and
native record pins; jobs/extraction/demos/graph/usage/plan-blocks/
probe-done/reattach-transport converted; record-adapter.test.ts deleted;
chat-transport tests pin the identity retry). 982/982, typecheck, lint,
build all green.
- src/agent/lh-tool.ts deleted: the zod-wrapped LhTool + toPiTool bridge
(z.toJSONSchema per call, clientQuestion flag, hand-rolled result mapping)
is gone. Tools are pi-agent-core native AgentTools now.
- New src/agent/tool-execute.ts owns what the wrapper added: the abort guard
(runWithAbortGuard: run signal + 30s per-call timeout + 250ms grace,
'Operation aborted' / timeout wording preserved), ToolInputError (the
retryable '- problem at path' format), result mapping (string verbatim /
object JSON / image parts, details rides the record's tool part output),
and questionResult() — placeholder + terminate:true (owner Option A:
belt of terminate + slim makeQuestionStopper with the clean-question
rule; maxTurns deleted from the teaching turn entirely per owner).
- All 11 tool files + tools.ts registries converted to TypeBox parameters
(typebox@1.3.7, pinned to pi-agent-core's copy). zod refinements
(quizInputProblems, askOptionProblems, live-nodeId) moved from schemas
into execute — the model sees the same retryable error shape.
- question-core + contracts/probe TypeBox-authored (single source shared
with the UI); tool-parts parse seam runs the boundary's full chain
(Convert → Check → Clean → refinement), fixing the live-smoke lockout:
pi's boundary Value.Converts correctAnswer: 17 → '17' but the record
keeps raw args, so a parse without Convert rejected a posed quiz
(invisible pending question, turn ended). Pinned with the real smoke
emission shape.
- Subagents converted: demo-builder/researcher AgentTool-native,
visual-maker/pdf-reader declaration-only pi-ai Tools with local execute
fns, extraction submit schemas re-authored in TypeBox (owner option 2:
zod defaults → Type.Optional + fill at the submit seam).
- Accepted delta, flagged: TypeBox-native schemas skip pi's JSON-Schema
primitive coercion fast path... (boundary still Value.Converts), and
prober-quiz unknown keys ride through (Value.Clean strips at display).
- Tests: 988 passing (was 967 at the recon baseline); tool-abort and
read-tool rewritten against the native seams; question-core/question-
tools/tool-parts/demos/web-search/graph-tools/plan-tree/probe-done/
tool-toggles converted to Value.Check/execute pins.
Live smoke (PORT=3004, real model): native ls turn streams end-to-end; a
real quiz pose (first attempt missing correctAnswer → tool-input-error →
model retry → posed) terminates the turn cleanly and persists pending.
Owner-diagnosed: the turn gate checked only model coordinates, so a
pinned slot whose provider key was deleted passed the gate and died
mid-stream masked to 'An error occurred.' The key is now gated where
the model is — by what would RUN (env-first keyValue truth, so an
env-shadowed key passes).
- core/settings.ts: the provider_key_missing envelope family (code,
provider, slots, human line naming the provider from AUTH_PROVIDER_KEYS
labels) + ProviderKeyMissingError + isProviderKeyRefusal matcher.
- chat-handler turn gate: refuses 503 before beginTurn when the resolved
conversation's provider key is gone (slots: [strong]).
- Job runner: same check via an injected dep (agent layer cannot read
the key store) — a keyless engine refuses the spawn through the
existing failure-shape convention, no phantom run.
- Extraction pipeline: refuseIfStagesUnready refuses BEFORE any batch
(first offender: extractor before auditor); the trigger skips info-
level, /api/graph/import surfaces the 503 envelope per file.
- Tests: 8 new pins (turn gate both ways, runner refusal, pipeline
first-offender + pass-through, envelope factory, matcher); the three
turn-driving suites seed OPENROUTER_API_KEY like any real setup.
- Doc fix: stale EXTRACTION.chunkTokens comment in the manual-extract
route (left over from eed4b4e).
Known residual (pre-existing, flagged for master): the manual extract
route logs BOTH config refusals as console.error (fire-and-forget path);
the coverage ledger keeps everything uncovered either way.
Owner-locked extraction-UX redesign (dispatch 2026-09-24), part 1 (data
layer; the UI chunk follows).
THRESHOLD -> SETTING
- config.ts's EXTRACTION.chunkTokens dies; the default moves into
FACTORY_SETTINGS.graph.extractionChunkTokens (4096, the old constant's
value). GraphSettings gains the optional field, Required at resolution
via the standard ?? FACTORY pattern — pre-redesign records resolve the
default cleanly.
- FLOOR 500 (the implementer's call, documented on the field): every run
pays the graph-carrying prompt overhead out of the write channel's
money — a smaller budget fires penny-batches that cost more per
extracted token than the content is worth. The sanitizer CLAMPS up
(bd68486 clamp-never-reject discipline: fractions floor, junk drops);
the settings PUT REJECTS below-floor with the rationale in the message.
No ceiling: a large budget just delays extraction — the user's stated
cost/thoroughness tradeoff, not a misconfiguration to police.
- The trigger (turn-service.triggerExtraction) takes the budget as a
parameter — chat-handler passes effective.graph.extractionChunkTokens,
the fresh per-turn read, same as every other gate.
- The pipeline's BATCH side threads through the same number:
ExtractionPipelineDeps gains an OPTIONAL getChunkTokens getter with the
factory default (Teacher's 24 test fixtures stay untouched; the two
call-site lines inside extraction.ts are the only cross-band edits,
flagged). chat-handler + backfill/import-notes wire the live
effectiveSettings — trigger and batches share one budget, always.
PER-SESSION COVERAGE READ
- sessionCoverage(chatId) (server/chats.ts): the chat view's ring-meter
block — totalMessages/coveredCount/uncoveredCount/coveredTokens/
uncoveredTokens (the same coverageOf derivation /api/graph/coverage
serves, scoped to one chat), running (chat-kind extraction registered
for this id), and lastFoldedAt (the chat's newest coverage row's fold
— a cheap single pass; null before any fold). GET /api/chats/[id]
carries it as coverage: {...}; unknown chat -> null (the route 404s
first).
- /api/graph/coverage (the all-sessions list) compiles untouched — the
per-session read is a scoping of it, not a supersession; the UI's
sidebar/ring chunk decides which surface it consumes.
Tests: factory/sanitizer pins (clamp-up, fraction-floor, junk-drop),
resolution pins (absent -> 4096, stored 2000 wins), settings-body pins
(valid/absent/fraction/below-floor 400/junk 400), sessionCoverage pins
(derived split, newest-fold scoping, cross-session isolation, running via
the registry, null on unknown chat). 893/893.
Part 1 — the ledger + remnants interlude (master-dispatched, owner ruling
2026-09-23: nothing configures the graph beyond chat models):
- The embedding ledger surface is gone: readEmbeddings/writeEmbeddings
ports, the EmbeddingsLedger contract, EMBEDDINGS_FILE (config, store
impls, graphRevision source list), and graph-deletion's vector-pruning
step. On-disk data files stay untouched (inert history).
- The extraction embedder seam dies wholesale: ExtractionPipelineDeps
drops the embedder dep (chat-handler + backfill/import-notes stubs go
with it), the 'embedding' phase leaves ExtractionPhase, the usage
capture/persist helpers die, and stages.embedding leaves
ExtractionRunUsage (historical records keep their shape — reads are
lenient). Extraction is two stages again, like the local-embedder era.
- Consumer-less remnants: latestQuizOutcomes + QUIZ_EVENT_KINDS,
clusterContentId, matchClusterLabelings + Community + ClusterLabeling,
the readClusterLabels/appendClusterLabel port seams + store impls +
LEDGER_FILES.clusterLabels. NamingRunUsage/readNamingRuns STAY
(standing ruling — the cost pill renders historical naming spend);
clusterHue + MIN_CLUSTER_SIZE stay (live).
- The 'labeler' role dies: BACKGROUND_MODEL_ROLES, ROLE_SLOT, ModelRole,
the catalog pin projections, and the UI roster strings. RolePins keys
shrink with the union.
The notes birth fallback (owner-approved 2026-09-23, surfaced by the
interlude's live smoke: production notes imports create provenance-EMPTY
nodes (importNotes passes messages: null — the file's anchor is the
mapping line), so the provenance-only birth rule left every imported
file unclustered and invisible to the prober's session list, against the
locked file-birth ruling. A provenance-empty node now births under its
FIRST mapping line's chatId (ledger order); only notes-born nodes ever
hit the fallback (chat nodes always carry provenance; a re-encountered
node's link-append re-opens the provenance rule). Deletion-safe: mapping
lines of deleted chats are stripped alongside their provenance.
Live-verified on port 3004: a fresh notes import clusters under its
src_ id with the file path as the label, end to end with no embedder.
845/845 tests, typecheck/lint/build green.
EOF
)
The owner's zero-configuration ruling (2026-09-23): nothing configures
the graph beyond chat models. The prior-knowledge injection — its budget
dial, its runner, its marker part, its dock/dispatch entries — is
removed across all bands; the prober now reads the learner's persistent
knowledge map on demand through two gated tools instead:
- list_sessions: every birth session, names + kinds + counts + dates
(graphSessionIndex, Platform's core seam)
- list_propositions(session): what was born there + the session's own
latest understanding level + that evaluation's date
(graphSessionPropositions) — names/levels/dating ONLY, no statement
text (owner ruling: the map's organization is the prober's business,
its prose is not)
Registered via createProberTools when chat-handler passes graphStore
(the graphToolsEnabled gate decided server-side, per-turn fresh); the
prober doctrine gains the map-review nudge at session start.
Also: runner's JobKind special-casing collapses (KIND_ROLE/KIND_RUNNERS
are plain records again), semantic-clusters.ts dies with its test, the
embedder dep's doc tells the wave-7 stub story, and ports.ts's
readClusterLabels/readNamingRuns surface stays untouched for Platform's
ledger-port interlude.
848/848 tests, typecheck/lint/build green.
Wave 7 chunk A (950ddfd) replaced semantic clustering with birth-session
clusters labeled by session names — the LLM namer became dead code, but
extraction still called it, minting naming runs nobody reads on every
merge. Real model spend wasted per extraction.
Removed (agent band):
- src/agent/subagents/cluster-namer.ts — the naming sub-agent, deleted
whole (refreshClusterLabels, clusterNamingSettled, the naming loop).
- extraction.ts: both post-merge garnish blocks (chat + notes paths),
the labeler test seam on ExtractionModels, and the import. The labeler
role's only consumer is gone; the 'labeler' entry in
BACKGROUND_MODEL_ROLES/ROLE_SLOT (core band) is now consumer-less —
flagged to Platform, not crossed.
- scripts/import-notes.ts: the settle-before-exit line + the named-
clusters count in the summary.
- tests: the cluster-naming describe (naming + failure-non-fatal tests)
and the dead CLUSTER_* fixtures; the labeler-less BYOK test repurposed
as the kill pin — a merge mints NO naming runs and NO label records.
NOT touched (flagged, outside the agent band): matchClusterLabelings
(core/clusters.ts), clusterContentId (core/graph.ts — now consumer-
less), the appendClusterLabel/readClusterLabels port seams + store
impls + ClusterLabeling type (store/core band), tests/clusters.test.ts's
matchClusterLabelings describe. Cost-summary's naming stage untouched
per Platform's ruling (historical spend display reads readNamingRuns,
which stays live).
Gates: typecheck, 921/921, lint, build.
Platform's partial-spend interlude made embedNewNodes fail BY RESULT:
landed vectors + partial usage ride the return value instead of a
throw. The extraction-side doc comments still claimed 'failed embeds
contribute nothing' — now they state the pay-once truth: skipped
(unconfigured) embeds carry no usage, failed ones carry the landed
batches' steps, and a mid-run failure leaves a three-stage run's
partial embedding spend on the ledger. The call-site comment says
clustering runs with what LANDED (not 'without the new nodes'), and
recordEmbedFailure's doc marks the catch belt-and-suspenders for
injected embedders that still throw (production no longer does).
DELETE /api/chats/[id]?mode=preserve-graph|delete-nodes (mode required):
- preserve-graph: only extraction bookkeeping dies (coverage rows,
legacy checkpoints) — graph ledgers byte-untouched
- delete-nodes: the chat's ledger traces die. Node survival = surviving
provenance point / mapping line (notes imports carry empty
provenance) / surviving event. The creator's aliases + provenance are
stripped from a surviving node's create line (owner ruling
2026-09-23: name/statement/createdAt are the shared identity);
orphan nodes, their edges, events, mappings, positions, embedding
vectors and cluster members die; cluster labels stay (immutable)
Gates: 404 unknown, 409 turn_in_flight / jobs_running (with jobIds),
503 extraction_winding_down after a 10s abort-and-poll; the purge is
one sync block (ledgers → jobs → chat record last). rewriteLedgers is
atomic (tmp + rename). graphRevision (max mtime over graph data files)
rides the activity feed + the response as the change stamp.
Anti-resurrection: the chat pipeline re-checks chat liveness at each
merge — a run that slips past the wind-down aborts with nothing merged
or checkpointed; its partial spend dies with the chat.
Owner decision 2026-09-22: 'it doesn't make sense to have any default
model configured anywhere; the application is bring your own keys.'
config.ts: MODELS deleted; ModelRole becomes an explicit 9-role union;
DEFAULT_THINKING_ORDER (medium, low, high, minimal, xhigh, max) bakes
the unset-level default — first entry the model supports; explicitly
set levels keep pi's native nearest-clamp.
core/settings-resolution: the factory layer dissolves — the chain is
role override → slot → NOTHING. resolveSettings(record) takes no
factory. Empty chains resolve to null/undefined; a thinkingLevel-only
slot is inert (stored, never run). resolveConversationSpec: chat.model
override over the strong slot, null when both empty. resolveRoleSpec
spans all nine roles for the job/extraction engines.
Refusal envelope (core/settings.ts): modelsNotConfigured(missing) —
{error, code: 'models_not_configured', missing: [slot tiers]} +
ModelsNotConfiguredError with .envelope. Per-need gates, each honest:
- chat-handler: universal turn gate (new + continuing turns) returns
503 before beginTurn — no liveness, no write-ahead persist, no graph
trigger before the gate fires.
- runner.runJob: unresolvable engine lands the job as error with the
envelope wording; no phantom loop, no result steer.
- extraction: refuseIfStagesUnconfigured throws before any stage
(turn-service info-skips so coverage retries next turn; the import
route surfaces per-file code+missing); labeler-less runs skip
cluster naming with a console line and still merge.
Write side: slot entries must carry a model — level-only slot writes
are 400s (absent slot key = clear). piEngineForSpec resolves unset
thinking levels via DEFAULT_THINKING_ORDER; piEngineFor now throws
honestly; thinkingLevelFor/modelIdFor deleted; teacher-agent, pi-run,
visual-maker, pdf-reader all read engine.level.
model-catalog: defaults/slots/rolePins null when the chain is empty
(contracts nullable) — no phantom ids in payloads. Stores drop
MODEL_PINS entirely.
Tests: pin-leaners converted to explicit injection (per-chat model
pins on scripted chats — exactly what the setup flow writes; faux
engines for agent-construction tests). New coverage: the 503 gate
envelope, the runner refusal, the extraction refusal + labeler skip.
Gates: typecheck, lint, 815/815, build.
Chunk B, item 3 of the search port + slots wave (2026-09-20): the flagged
gap closed — ExtractionModels deps were engine VALUES, keeping
extractor/auditor/labeler config-pin-bound while the job runner went
slot-native. The pipeline now takes the runner's exact treatment:
- ExtractionPipelineDeps.engineFor: (role) => PiEngine — resolved FRESH
per stage call, so a settings PUT between stages lands. Direct test
values (extractor/auditor/labeler) remain the priority seam, checked
FIRST; absent values fall to the chain; absent chain → config pins
(unchanged).
- All six stage call sites (chat + notes extract/audit, the cluster
namer x2) resolve through stageEngine(role, direct).
- chat-handler's extraction composition injects the full chain —
piEngineForSpec(resolveRoleSpec(role, readSettings(), MODEL_PINS)) —
same wiring as the job runner.
Chunk B, item 4 (session-level slot-anchored model picks): VERIFIED
already complete from Platform's model-slots wave — the conversation
engine chain (chat-handler) anchors on effective.models.strong, the
picker's default surface (catalogDefaultsFor) falls back to strong,
both pinned in tests. No phase split (user decision, 2026-09-20).
Capability-filtered pickers live in the settings UI (main 343be10).
Tests: engineFor fills the stages end-to-end (both propositions merged
through the chain, consulted in pipeline order) + the direct-value
priority pin (engineFor that throws when consulted).
Gates: typecheck, 795/795 tests, lint, build — all green.
Model slots wave (2026-09-20), chunk B:
- settings: new `models` section {strong, cheap, vision} slots (each
field independent, same shape as a role override); ROLE_SLOT maps all
nine roles to tiers (strong=prober+teacher, cheap=researcher/
demoBuilder/extractor/auditor/labeler, vision=visualMaker+pdfReader);
resolution chain per role: override → slot → factory pin.
- config: ModelSpec gains provider (all pins openrouter, the BYOK
default); MODEL_PINS in stores.ts now spans all nine roles (the strong
slot's factory rep IS the prober pin).
- engine: piModels() registers EVERY builtin provider catalog;
piEngineForSpec(resolved spec) is the new engine factory (pi-ai
resolves per-provider keys from env itself); piEngineFor stays as the
config-pin fallback. chat-handler resolves the conversation chain
(chat override → strong slot → pin) and injects engineFor into the job
runner, closing the old gap where per-role overrides never reached
job engines.
- /api/models: models federate ONLY keyed providers (providers surface
with keyPresent, env-wins, fresh per call); slots (resolved tier pins)
join rolePins; catalogDefaultsFor inherits the strong slot; the models
settings section gets the same catalog gate as chat.
- pricing: PRICES/CONTEXT_LIMITS uniformly keyed provider:id
(regenerated from ALL pi-ai catalogs, 1354 keys); usage stamps
(TurnUsage, JobUsage, extraction stages) switch to provider:id —
owner decision: no legacy compat, old stamps render unpriced.
- Cluster namer bills the money channel now: per-step usage + model
captured via onStep, persisted as NamingRunUsage records to
data/graph/naming-runs.jsonl (idempotent per startedAt; failed/aborted
runs persist too — money spent either way). callUsage promoted to
extraction.ts module scope for reuse.
- core/cost-summary.ts: the pure fold behind the pill — chat runs,
notes-source runs, and naming runs into stage totals (extraction /
audit / naming) + newest-first recent runs. Dollars INFERRED read-time
via prices.ts, never stored; unpriced models excluded-but-counted
(render '—', never a silent zero); tokens always sum.
- GET /api/graph/costs (src/server graphCosts composition, thin route).
- Graph view: bottom-left cost pill (cumulative dollars headline) +
expandable breakdown panel (stage rows with tokens secondary, recent
runs with time/title/merged/cost); fetched on mount, refetched when a
pipeline run finishes; merged into one bottom-left stack with the
running-pipeline pills.
- Tests: pure-fold suite, naming capture wiring (full importNotes ->
naming path), store idempotency + strict reads.
THE MESSAGE IS THE UNIT OF COVERAGE (vision-extraction-coverage.md): a
per-chat append-only ledger (coverage.jsonl) records which messages the
pipeline has extracted, by content fingerprint over exactly what the
extraction view sees (one shared line builder — core/coverage.ts); the
uncovered remainder is DERIVED at read time, never dual-tracked. A quiz
answer merging into the asking message re-opens only that message;
plan-block data-part attach never re-opens anything.
- The trigger (turn-service) replaces every-3-quiz-answers with
uncovered-tokens >= EXTRACTION.chunkTokens (4096) — quiz-less sessions
extract too, content size is the honest signal.
- runExtraction is now chunked: token-budgeted batches of the oldest
uncovered consecutive messages (a tool call + result never split), each
merged batch checkpointing its messages' coverage atomically — an abort
or error loses at most the in-flight batch; merged batches stay covered.
- Observation refs are batch-tagged (obs:<batchTag>:<sourceId>): re-runs
of the same batch dedupe; a re-opened message's fresh understanding
lands as a new event. Fixes the old obs:p1 position-collision.
- New API: GET /api/graph/coverage (exact per-session derived coverage +
running flags, most-uncovered first) and POST /api/graph/coverage/extract
(manual run for one session, 409 while in flight).
- Graph view chrome: the coverage button opens a panel with every
session's covered/uncovered tokens and a per-session Extract button —
the manual answer to aborted runs, errors, and dormant tails. Renders
even when the graph is empty: an empty graph with uncovered sessions is
exactly when it matters. Polls only while open.
- Notes imports keep the whole-file checkpoint path (coverage undefined).
- Tests: pure-core coverage suite (11), chunked-pipeline suite (3:
multi-batch + no-op, abort-keeps-earlier-batches + remainder re-run,
re-open alone + data-part invisibility), panel render suite (5);
pendingQuizAnswers and its tests deleted as dead code. 486/486.
Live-smoked on real GLM: manual extract of "teach me modal realism" ran
one batch, wrote 4 coverage entries, panel flipped to "1.3k covered ·
complete" with the button disabled, graph refetched (nodes 11), usage
persisted with real stats.
Louvain only ever found the reasoning chains (the graph is disconnected
by nature — every connected component became its own cluster) and
edge-orphans could never join anything. Clusters are now RELATED TOPICS:
propositions grouped by meaning.
- Embedder (src/server/embeddings.ts): in-process transformers.js v4
running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download),
MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs,
L2-normalized so cosine = dot. Model/dim mismatch voids the ledger
(full re-embed). Injected at the composition root — the agent layer
never imports the server layer (architecture law).
- Ledger (data/graph/embeddings.json): model + dim + vectors + the
persisted cluster partition. Reclustering runs at the END of every
embedding batch (post-merge hook, both front doors) — never per read
(the agglomerative pass is ~1.5s at 338 nodes).
- Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage
agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers
stay singletons by default; attachStragglers is the all-in toggle.
Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans
land in real topics, 10 stragglers.
- Consumers (snapshot, settle, namer, replay) read the persisted
partition — one cluster concept everywhere; core-survival label
matching unchanged (20/26 topics kept labels).
- Louvain retired: detectCommunities + graphology* deps deleted
(graphology-louvain had leaked into the client bundle since the
clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed.
- scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s
including the one-time model download).
The 'virtual node' realized as a centroid force: two allocation-free
O(n) passes per tick (membership resolved once at sim creation;
Float64Array sums). Measured indistinguishable from force-off at 338
nodes (271ms vs 272ms per 400-tick settle).
- createGraphSimulation: SimOptions.clusterPull + linkStrength +
velocityDecay (physics controls plumbing); degree-scaled link
stiffness so linkStrength=1 is exactly d3's default
- settleGraphLayout split into layout-settle.ts (it computes the
Louvain partition for the pull); layout.ts stays browser-barrel-pure
- barrel no longer re-exports clusters.ts: graphology + louvain are
fully out of the client bundle (verified against built chunks)
- replay passes the per-batch partition; CLUSTER_PULL_STRENGTH=0.15
tuned by sweep: cluster spread 63->36 world units, centroid
separation and total span unchanged (no blob merge, no collapse)
The layout now EVOLVES instead of reshuffling on every extraction. A pure
core module (src/core/layout.ts) warm-starts forceAtlas2 from the positions
ledger (data/graph/positions.json, read-modify-write like sources.json,
updatedAt stamp for future live-physics conflict handling), re-blends every
previously-positioned node toward its anchor each iteration (retention 0.85
- old placements stay recognizable, strong forces still win), seeds new
nodes near their positioned neighbors, and seeds orphans on the rim outside
the core so they read as unconnected knowledge.
- Port/store: GraphStore.readPositions/writePositions; snapshot carries
positions when the ledger exists (optional - fresh graphs unchanged).
- Extraction pipeline settles synchronously after every merge (chat and
notes paths), before usage persist - the next /api/graph fetch already
reflects new nodes.
- View: the ledger IS the layout; the local one-shot FA2 pass survives
only as a fallback when no ledger exists.
- scripts/replay-layout.ts: simulates organic growth by replaying the
current 338-node graph one imported file at a time (real import order,
edges activate when both endpoints exist). Real run: 338 positions in
0.3s, 90-degree-0 orphans collecting at mean r=338 vs connected r=194.
- Also restores the cluster region fills (hull background colors) that
6692b19 had removed on a misreading of "no glow".
Live-verified: fit zoom derives from the ledger bbox, pickAt aligns with
ledger positions at every probe, hover/click/detail-card work, 408/408
tests, typecheck/lint/build green.
Future live physics stays cheap by design: the same pure function is the
merge-time settler and the future client simulator; a pinned node is just
an anchor with full retention.
Core-survival labelings (vision section 1, DECIDED 2026-09-13):
ClusterLabeling records in a new append-only ledger
(data/graph/cluster-labels.jsonl), immutable, id = clu_ + sha256 of the
sorted core so re-naming races and re-runs dedupe at the store seam.
matchClusterLabelings (core/clusters.ts, pure): a community keeps a
record's label while at least half its core survives in it; both sides
single-successor (a split keeps one name; greedy by overlap, ties to
the newer record).
The namer (src/agent/subagents/cluster-namer.ts): ONE singular cheap LLM
call (new 'labeler' role, glm-flash/low) names every new/changed
community at once, given member names + statements (capped at 12
members per cluster). refreshClusterLabels runs POST-EXTRACTION
(user-decided) from both front doors, fire-and-forget — naming failure
never fails the run; refreshes serialize behind a chain so parallel
imports never mint duplicate labels. import-notes.ts awaits
clusterNamingSettled before exit.
Payload: GraphSnapshot.clusters (>= MIN_CLUSTER_SIZE 3) joins detection
with labelings at read time — claimed clusters carry the record's label
and id; unlabeled ones carry a member-hash id and null label. Detection
is deterministic so no memo in v1 (ms at current scale; revisit at
vault scale).
Tests (372/372): matcher half-rule/split/merge/greedy/tie cases; naming
end-to-end through importNotes (one call, one record, claim on
re-refresh), naming failure non-fatal; snapshot derivation with
labeling join; 11 test pipelines now pin an explicit labeler seam.
Diagnosis from the real 25-file notes import (user-pasted session logs):
almost every run hit 'Validation failed for tool submit_findings/review:
edges must have required properties edges' — the model (glm flash, low
thinking) omits top-level fields when they're empty, and pi-ai's
structural JSON-Schema validation runs BEFORE the zod layer, so each
omission cost one of the extractor's maxTurns 4 (full payload
regeneration) and occasionally burned out the run entirely (the two
FAILED imports).
The model is technically wrong, but the schema provokes it:
required-but-usually-empty arrays are a known weak-model anti-pattern.
- lhTool's model-facing declaration now converts with io: "input"
(the declaration describes what the MODEL sends): .default() fields
drop out of required. No-op for teacher tools (none use defaults).
- findingsSchema.edges / auditSchema.dropped: .default([]) — omission
means "none"; attribution: .default({quiz:{}}) for quiz-free sessions.
- Prompts hardened in all three agents: include every top-level field,
never omit because empty.
Structural probe: validateToolArguments now passes the exact pasted
failure shape; zod re-parse in execute applies the defaults. 355/355
tests, typecheck, lint, build green. (lh-tool.ts is the shared engine
seam — flagged to master: io:"input" is the correct mode for every
model-facing declaration, teacher path verified unaffected.)
The same extractor→auditor→merger pipeline now runs over raw personal
notes in arbitrary formats (vision §1, DECIDED 2026-09-13):
- NotesSource records (data/graph/sources.json, deterministic id from
path hash → idempotent re-imports) play the chatId role in mappings,
checkpoints, and observation events; usage lands on the source record
(appendSourceUsage, idempotent per startedAt, kept across upserts).
- mergeFindings is chat-store-free: callers pass MergeSource
{messages, foldedAt} — sessions pass the chat record, notes pass
messages:null (no quiz-event rebuild; a deterministic rebuild needs
structured quiz tool parts).
- Second extractor prompt (NOTES_EXTRACTOR_INSTRUCTIONS) for general-
purpose extraction; understanding judges the AUTHOR's grasp. Auditor
wording input-agnostic. extract/audit share one generic runSubmitAgent.
- graphSnapshot.fileSources (srcId→filePath); notes sources render as
static path entries in the detail card (FileTextIcon, never a link).
- One shared pipeline instance (exported from chat-handler): same
single-flight registry, live status chips (label = file path), usage
accounting, and true abort for both front doors.
- Transport: POST /api/graph/import (multipart, 3-wide worker pool) +
Import notes… button in the graph view; scripts/import-notes.ts for
batch runs (concurrency 4).
- real-data-extraction/ gitignored (raw import sources never committed).
355/355 tests; typecheck, lint, build green.
The Vercel AI SDK survives only as Band D (useChat client + UIMessage
record types); @openrouter/ai-sdk-provider is uninstalled.
- src/agent/models.ts: PiEngine {model, streamFn} via createModels +
openrouterProvider subpath; thinkingLevelFor maps config effort none->off
- src/agent/lh-tool.ts: neutral LhTool (zod authoring, optional execute,
clientQuestion flag); toPiTool converts via z.toJSONSchema and re-runs
zod inside execute so refinements ride the error-result retry path
- src/server/pi/record-adapter.ts: UIMessage records -> pi Messages
(dangling tool calls get synthesized interrupted results)
- src/server/pi/ui-stream.ts: SSE bridge emitting AI-SDK-shaped chunks so
the useChat client parses unchanged (continuation id rule, question
tools never emit output, tool-input-error for bad authoring)
- teacher-agent.ts: stateless per-turn pi Agents; shouldStopAfterTurn
stops only when every question-tool call resolved cleanly (errors retry)
- chat-handler.ts: buildAgent(outgoing) factory seam (stateless doctrine);
assertValidUiMessages owns the retired SDK validation gate
- sub-agents via shared runSubagent (pi-run.ts); extraction stages stop
after a clean submit; visual-maker keeps its hand-rolled loop over
direct streamSimple calls; aborted model calls bill no usage
- tests: all model seams ported to pi-ai faux provider (scripted
FauxResponseSteps, agent factories); handleChatRequest second param
is now an agent factory
Gates: tsc, lint, build, 305/305. Live GLM smoke (disposable driver,
deleted): prober streaming + persistence, question seam, turn-2
continuation same message id, per-turn real token usage - ALL PASS.
Data wiped at the pivot (chats/graph/jobs, .gitkeeps kept).
Extraction is real spend and background work; the dock needs to see
and stop it. User decisions: persist all cost data (chat record),
per-stage usage granularity, and TRUE cancellation for abort.
USAGE CAPTURE (live + persisted):
- extraction.ts: per-call onStepFinish captures each billed step's
usage (SDK LanguageModelUsage -> CallUsage: input/output/cache
read+write/reasoning) into the registry's live per-stage usage
(extractor/auditor; the merger is deterministic — no model call).
- /api/graph/status runs now carry usage (model + steps per stage);
cost/context folds are the consumer's (core/prices.ts — nothing
stores dollars).
- ChatRecord.extractions: append-only per-run usage log (mirrors the
turns log): startedAt key, at, durationMs, merge stats when the
merger ran, stages. Persisted on success AND failure — a failed run
still spent money. Idempotent per startedAt; the list payload
(ChatSummary) never leaks it.
- EmissionStats moved to contracts/graph.ts (its contract home);
core/graph.ts re-exports for existing importers.
- Verified end-to-end against the real pipeline (backfill one chat):
per-step token counts, reasoning tokens, and the auditor's growing
prompt-cache reads captured and persisted.
TRUE ABORT (the dock's abort button, user decision):
- RunningExtraction owns one AbortController per run, shared across
phases; abortExtraction(chatId) fires it and returns whether a run
was aborted (idempotent no-op otherwise). The pending model call
receives the signal — the provider cancels the request, spend
stops. throwIfAborted() at phase boundaries ends runs aborted in
sync gaps.
- Abort is a deliberate stop, not a failure: same ledger discipline
(nothing merged, nothing checkpointed, partial usage persisted) but
the run reports zero emissions instead of throwing. Exported via
src/server/observability.ts for master's /api/activity/abort route.
- Tests: mid-run abort (hanging mock that rejects on the signal),
live-accumulation + persisted-record assertions, store-seam
idempotency, and the summary-payload leak guard.
amosblomqvist