9f6401908798d8f1ac875a907327bbfee526de0e
30
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 amosblomqvist
|
af666aaa3f |
physics: bundles join their cluster; depth runs over the logical graph
Owner-approved brief (orchestrator, physics arc): two laws.
1. IMP-NODE CLUSTER ASSIGNMENT. Imp nodes were unclustered (the partition
runs over stored propositions only) — no centroid pull, outside the
painted hulls, bundles sitting on the periphery pointing outward.
Now: a derived imp node's cluster is a function of its derivation —
MAJORITY among its PREMISES' birth sessions (owner: premises, NOT the
conclusion), ties to the chain's OWN birth (its provenance's first
point — the asserting session; deterministic, coincides with the
unanimous common case). Imp nodes are always re-derived, never
birthed, so the assignment must be exactly this.
Seam chosen: core/session-clusters.ts exports impClusterKey + a
shared attachImpToCommunities that mutates a communities list's
members IN PLACE. graphSnapshot attaches AFTER the hull filter (a
2-proposition community + its bundle never resurrects a hull — the
filter counts propositions); settleGraphLayout attaches to the
unfiltered partition (settle pulls every community; hulls are a
canvas-only concern). Both cluster-membership consumers — the sim's
centroid pull and buildRegions' rasterization — read it through the
members lists that went through the one helper, so the resting
ledger and the live sim agree on where bundles belong.
2. DEPTHS OVER THE LOGICAL GRAPH. inferenceDepths ran Tarjan over the
DRAWN edges (premise→imp→conclusion), inflating every multi-premise
conclusion one hop (depth(imp)+1 instead of depth(premises)+1) and
ranking the geometry point itself. Now:
- deriveView additionally emits logicalEdges — every chain contracted
to direct premise→conclusion arrows over PROPOSITION ids only
(per-premise, deduped — a conclusion's rank is max over premises,
which a single arbitrary premise edge could not express).
- GraphSnapshot carries logicalEdges; createGraphSimulation /
incrementalLayout accept a logicalEdges option: the depth
machinery (orphan detection AND the rank pass) reads it, springs
keep the drawn edges. Absent = edges (legacy behavior).
- Imp nodes join the ORPHAN-EXEMPTION mask explicitly — never pinned
to a level, keep the full regular forces (gravity, cluster pull,
springs), settle naturally between premise and conclusion levels;
the rank map is computed over propositions only, so arg_ ids are
absent from it by construction.
Wiring: settle + canvas (CanvasData.logicalEdges → buildSim) +
replay-layout all pass the logical set.
EXTRA FIX (in-slice discovery, one line, graph-server domain):
server/graph.ts graphClusters — the practice picker's /api/graph/
clusters — partitioned snap.NODES (the canvas image, imp nodes riding
it since chains-primary), leaking arg_ ids into nodeIds: pickable ids
that silently vanish inside chainsForSelection while nodeCount lied.
Now partitions snap.propositions — same purity law as the chain
table's 767a559 fix.
Tests (+13): impClusterKey pins (unanimous; cross-session majority; tie
→ chain's own birth, not the conclusion's); attach pins (members gain
the imp id, sorted; single-premise attaches nothing — no node exists;
dangling mints nothing; home absent from the list stays unattached);
snapshot pins (imp id rides cluster members to both consumers; the hull
filter counts propositions); logical-depth pins (multi-premise
conclusion at premises'+1 NOT +2; imp absent from the rank map);
depth-force exemption pin (rows pin crisply, arg_x floats unpinned —
old behavior stamped it onto the conclusion's row); deriveView logical
arrows (per-premise, deduped, proposition ids only).
|
||
|
amosblomqvist
|
719f7f9934 |
rendering arc: the imp node becomes the argument's geometry, not a fake proposition
Owner-approved brief (orchestrator): the canvas stops drawing implication nodes as first-class nodes and starts drawing them as what they are — the point where co-premise lines converge. - deriveView (core/graph.ts): single-premise chains collapse to ONE direct premise->conclusion arrow — no imp node minted (a chain is always N-in/ 1-out; one premise is the degenerate case). Multi-premise chains keep the imp node + fan. Dangling guard unchanged. - physics (core/layout.ts createGraphSimulation): per-link rest length — a link touching an arg_-prefixed endpoint gets HALF the base link distance, default or user-tuned alike; everything else about imp-node physics citizenship unchanged. - canvas (components/graph-canvas.ts): imp nodes are PURE GEOMETRY — invisible (no disc, no label) and unpickable (excluded from pickAt's hit-test projection). Edges INTO an imp node: headless, shaft runs to its CENTER (toRadius 0, no head triangle). Edges OUT: shaft starts at the center (fromRadius 0), arrowhead lands on the conclusion's disc edge. Single-premise direct arrows render as today's plain arrows. - comment repairs (same commit): the two stale double-shaft claims (CanvasEdge doc + the draw-pass block) rewritten to the truth — plain single shafts; bundles are headless converging lines + one arrowed out-edge; no double-shaft glyph exists anywhere. - tests (+9): deriveView pins (single-premise collapse, multi-premise fan, shared-conclusion distinctness, dangling drop); imp-link half- distance settle pins; arrowGeometry zero-radius laws + isImpNodeId pin; graph.test.ts's snapshot invariant now counts only MULTI-premise live chains (its fixture's single-premise chains mint zero imp nodes); layout settle fixture updated for the collapse. |
||
|
amosblomqvist
|
8d72de596a |
fix: chain table reads the stored propositions, not the derived canvas image
The owner's live test surfaced it: after importing notes, the chains view listed implication nodes as bogus singleton "chains" — and each carried the same arg_ id as the real stored chain, tripping React's duplicate-key error (chain-table.tsx:372). Root cause: graph-view passed snapshot.nodes — the derived canvas view (propositions + implication nodes) — into ChainTableOverlay, whose chainsForSelection turns every unchained node into a singleton chain. An implication node is never a chain member (members are proposition ids), so every imp node minted a bogus singleton with a colliding id; its row was even selectable, leaking arg_ ids into practice input. Fix (owner-approved shape): GraphSnapshot gains `propositions` — the stored-truth node list riding beside `chains` for exactly this class of consumer. The chain table now receives snapshot.propositions; the canvas keeps snapshot.nodes (imp nodes are meant to be drawn). Practice pipeline unaffected (it already read getGraph, which is propositions-only). Test: the snapshot split pinned as an invariant in graph.test.ts — image = truth + one imp node per fully-live stored chain; truth never carries an arg_ id. |
||
|
amosblomqvist
|
4985b6f9fa |
chains-primary: stored chains replace typed edges; edges become a derived view
Owner rulings 2026-09-30: reasoning chains are the stored truth; the graph canvas renders a lossy derived image. No migration, no backcompat — old graph data was wiped. - contracts: GraphChain (arg_-minted id, premises, conclusion) replaces typed edges; GraphEdge survives only as the untyped derived arrow; GraphSnapshot carries stored chains; supports edges retired entirely - store: chains.jsonl ledger (edges.jsonl purged at load alongside legacy events.jsonl); readChains/appendChain seam; merge dedupes by (sorted premises, conclusion), re-assertions append under the same arg_ id - core: deriveView mints implication nodes (name ⟹, statement via formatChainLine) plus premise→imp→conclusion arrows; clusters stay proposition-only (birthSessionPartition); chainsForSelection now groups STORED chains (chain rides iff every member selected) — independent arguments for one conclusion never union; chainsToGraph/chainsToEdges die - extraction: extractor submits chains natively (expandFindings deleted); auditor receives propositions only; supports fan-out gone - deletion: full death of a proposition kills every referencing chain; arg_-prefixed node deletes kill that chain; chainsRemoved stats - UI: every arrow renders the ⟹ double shaft; supports toggle + thickness slider and detail-card edge sections deleted (card keeps name + proposition); chain table + practice pipeline read stored chains - tests: 1188 green across the new seams |
||
|
amosblomqvist
|
8355fbabdb |
delete the mappings ledger — per-job in-memory resolution + notes fingerprint guard
The persisted (chatId, sourceId)→node table was a shared namespace across independent chunk/batch jobs: each chunk's extractor mints local ids fresh (p1..pN), so later chunks collided with earlier ones — propositions silently skipped as 'already resolved' (coverage checkpointed anyway: permanent silent loss) and edges resolved onto the wrong nodes (the 'everything links to everything' mush). Owner ruling 2026-09-29: mergeFindings resolves THIS job's ids in memory, nothing persists between jobs; cross-job identity is the auditor's linkTo verdicts alone. - contracts: NodeMapping dies; NotesSource.contentFp — sha256 of the text at the last run, the re-import guard (unchanged re-import = TRUE no-op, no model calls, no usage; changed text re-runs; legacy unstamped sources re-run once) - merger: job-local resolution map, in-submit duplicate ids skip once, edges resolve extractor-local ids only - ports/store: appendMapping/readMappings, LEDGER_FILES.mappings, rewriteLedgers.mappings gone (stale data/graph/mappings.jsonl just stops being read) - session-clusters: birthSessionOf is provenance-only — the legacy mapping-line fallback for provenance-empty notes nodes dies - deletion: no mapping rows to strip; stats lose mappingsRemoved - replay-layout batches nodes by first provenance point - graph.test pins the new independence semantics (re-merge re-creates; duplicate ids within one submit resolve once); extraction tests pin per-chunk merges + the fingerprint guard |
||
|
amosblomqvist
|
3d9d20794c |
Auditor submits links ONLY — the edges argument is gone (owner, 2026-09-28 evening)
Hypothesis under test by the owner: coherent extractions turn to mush after merging because the auditor minted too many bridging edges of its own — inference, not identity, and out of the auditor's scope. The graph now connects across sources solely through identified equivalences; relations between propositions come from the extractor alone. - auditSchema drops 'edges' (submit_review = links only); the submit-seam required-field doctrine reworded per tool - AUDITOR_INSTRUCTIONS: the BRIDGING EDGES section dies; deliverable is the identity verdicts only - AuditVerdicts drops edges; ExtractedEdge endpoints are extractor-local ids only; mergeFindings stops consuming verdict edges and its resolve() drops the raw-node-id branch (dead — the extractor never sees graph ids) - extractor untouched: submit_findings keeps propositions + follows/ supports edges, required as before - tests: 17 submit fixtures + 9 typed verdict literals slimmed to links - docs: architecture.md division-of-trust + pipeline diagram updated |
||
|
amosblomqvist
|
f11b1664dc |
verdicts-only auditor: submit_review is {links, bridging edges} — the findings echo dies
Owner ruling 2026-09-28: the auditor has no control over the extractor's output and never alters it, so re-emitting the full findings per submit was pure payload cost — the retry collapse the schema slim attacked, halved again. submit_review is now links (local proposition id -> existing graph node; EQUIVALENCE BIAS carried in the linkTo description) + bridging edges ONLY; the merger joins extractor findings + verdicts. mergeFindings(chatId, findings, verdicts, ...) — propositions come from findings verbatim, links join by id, edges emit extractor's then auditor's through one dedup-preserving emitter. The old echo form hid an accidental drop channel: an auditor that omitted a proposition from its echo silently dropped it from the merge. Verdicts-only removes it structurally — extractor output always fully merges (tests updated to full-merge expectations). AuditResult/AuditedProposition -> IdentityLink/AuditVerdicts; audit() logs links/bridges; batch view renders anchorless like transcriptView; auditor prompt rewritten to the verdicts-only deliverable; architecture.md division-of-trust updated. |
||
|
amosblomqvist
|
bd9694b0e4 |
slim the submit schemas: evidence and messages die; stringified-payload heal at the pre-validation seam
OpenRouter log diagnosis (owner, 2026-09-28): extraction time was
dominated by tool-validation retries. The failing call had the whole
propositions array emitted as ONE JSON-STRING ('propositions.0: must
be object'), a truncated placeholder artifact inside it, and
word-salad statements late in the array — payload-size collapse.
Field audit: 'evidence' was never persisted anywhere and its
faithfulness-checking consumer died with the fully-trusted ruling;
'messages' fed messageId granularity that NOTHING downstream reads,
and the model sent [] anyway (chat-level provenance in practice).
'aliases' stays (optional now): the auditor's find tool globs them.
Schema: propositions = {id, name, statement, aliases?}; edges =
{from, to, type}. Prompts lose the evidence/messages doctrine and
[id]-anchor guidance, gain a compact faithfulness line;
transcriptView renders anchorless. The auditor echo instruction
drops the evidence/messages mention, so its payload halves too.
Heal: healStringifiedArrays on the submit tool's prepareArguments
(pi-agent-core's sanctioned pre-TypeBox shim) parses a stringified
propositions/edges back into arrays — the LaTeX-restore doctrine:
heal weak-model wire behavior at the seam. Test drives the real
agent-loop seam with a stringified submit.
|
||
|
amosblomqvist
|
1b50013551 |
extraction cleanup: events ledger dies, markdown source view, auditor-as-linker (2026-09-27 rulings)
The graph becomes raw reasoning chains only. Quiz outcomes and understanding observations are no longer formalized; assessment moves to the future practice module. Events ledger removed end-to-end: LedgerEvent/LedgerEventKind/ UnderstandingLevel/ExtractionAttribution/AuditResult.dropped and snapshot nodeObservations deleted from contracts; quizEvent builder and the events step of mergeFindings gone; appendEvent/readEvents dropped from the port and store; LEDGER_FILES loses events.jsonl with a one-time purge of a stale file next to the nodes ledger; graph-deletion and the delete dialog lose event purge + quiz-answer impact counts; GraphSessionProposition becomes a pure birth list. Whole-session markdown source view: new core/session-markdown.ts is the shared formatting engine (id-anchored speaker blocks, quiz/ recall/question blocks, tool one-liners) — the extractor's prompt view and the future user-facing export render from one engine. Coverage fingerprints deliberately stay on the old line format (content-hash semantics; re-formatting never re-opens covered history). The whole session goes to the extractor — no phase cut. Auditor refocused as the identity linker: never sees the source; explores the graph via new auditor-specific tools in agent/tools/graph.ts (find — glob over name/statement/aliases; list_sessions; list_propositions with full statement text; no timestamps) instead of an injected full-graph dump; equivalence bias — paraphrases merge, only genuinely distinct propositions stay apart; no drop channel, the extractor is fully trusted; may draw bridging edges (typed follows/supports) between existing nodes and the batch, resolved by the merger with chat-scoped message-less provenance. Agentic loop, maxTurns 16. UI: the node detail card gains an edges section above the sessions list (follows-from / supported-by / what-follows / what-it-supports, rendered only when present, click-to-select neighbors); the understanding-note chips are gone. Prober list_propositions reworded to what a session covered — names only, no levels. docs/architecture.md + README updated to the as-built pipeline. Gates: typecheck, 1059/1059, lint, build. |
||
|
amosblomqvist
|
f3003d357a |
Wave D: native AgentEvent wire + client session rewrite (SDK purged)
Server: ui-stream.ts is verbatim pi AgentEvent passthrough (SSE data lines + [DONE]); only app-level events are the loop-level error and turn-settled terminal; onError carries raw provider text (owner ruling). Client: useChat + the AI-SDK transport replaced by components/session-core.ts (SessionModel: 409 identity-retry, optimistic c_ appends, absorbed() adoption, needsContinuation, shared consume for POST + reattach) and the session.ts hook; the golden-master grouping canon moved client-side into core/view.ts (projectView); scanners read the native record (phaseAt3Lh, danglingQuestionOf, graph toolParts, plan-blocks, runner, extraction). Coverage: the native message is the unit (fork 3) with the call+result inseparability bond rule; view-line-less messages (markers, error results) are invisible to coverage and never checkpointed. Deletes: chat-transport, projection, ui-message, their tests (ported to tests/session-core.test.ts), and the ai + @ai-sdk/react dependencies — the Vercel SDK is fully out of the tree. Wire pins in chat-flow rewritten onto the AgentEvent dialect (eventsOf parser). Gates: tsc clean, 994/994 tests, eslint clean, build green; live smoke on PORT=3004 (verbatim wire → native record). |
||
|
amosblomqvist
|
d1b46132b9 |
Wave 7 close-out: the embedding ledger seam dies; notes birth via mapping lines
Part 1 — the ledger + remnants interlude (master-dispatched, owner ruling 2026-09-23: nothing configures the graph beyond chat models): - The embedding ledger surface is gone: readEmbeddings/writeEmbeddings ports, the EmbeddingsLedger contract, EMBEDDINGS_FILE (config, store impls, graphRevision source list), and graph-deletion's vector-pruning step. On-disk data files stay untouched (inert history). - The extraction embedder seam dies wholesale: ExtractionPipelineDeps drops the embedder dep (chat-handler + backfill/import-notes stubs go with it), the 'embedding' phase leaves ExtractionPhase, the usage capture/persist helpers die, and stages.embedding leaves ExtractionRunUsage (historical records keep their shape — reads are lenient). Extraction is two stages again, like the local-embedder era. - Consumer-less remnants: latestQuizOutcomes + QUIZ_EVENT_KINDS, clusterContentId, matchClusterLabelings + Community + ClusterLabeling, the readClusterLabels/appendClusterLabel port seams + store impls + LEDGER_FILES.clusterLabels. NamingRunUsage/readNamingRuns STAY (standing ruling — the cost pill renders historical naming spend); clusterHue + MIN_CLUSTER_SIZE stay (live). - The 'labeler' role dies: BACKGROUND_MODEL_ROLES, ROLE_SLOT, ModelRole, the catalog pin projections, and the UI roster strings. RolePins keys shrink with the union. The notes birth fallback (owner-approved 2026-09-23, surfaced by the interlude's live smoke: production notes imports create provenance-EMPTY nodes (importNotes passes messages: null — the file's anchor is the mapping line), so the provenance-only birth rule left every imported file unclustered and invisible to the prober's session list, against the locked file-birth ruling. A provenance-empty node now births under its FIRST mapping line's chatId (ledger order); only notes-born nodes ever hit the fallback (chat nodes always carry provenance; a re-encountered node's link-append re-opens the provenance rule). Deletion-safe: mapping lines of deleted chats are stripped alongside their provenance. Live-verified on port 3004: a fresh notes import clusters under its src_ id with the file path as the label, end to end with no embedder. 845/845 tests, typecheck/lint/build green. EOF ) |
||
|
amosblomqvist
|
6b13583e96 |
wave 7: the prober reads its map — graph-context injection dies whole
The owner's zero-configuration ruling (2026-09-23): nothing configures the graph beyond chat models. The prior-knowledge injection — its budget dial, its runner, its marker part, its dock/dispatch entries — is removed across all bands; the prober now reads the learner's persistent knowledge map on demand through two gated tools instead: - list_sessions: every birth session, names + kinds + counts + dates (graphSessionIndex, Platform's core seam) - list_propositions(session): what was born there + the session's own latest understanding level + that evaluation's date (graphSessionPropositions) — names/levels/dating ONLY, no statement text (owner ruling: the map's organization is the prober's business, its prose is not) Registered via createProberTools when chat-handler passes graphStore (the graphToolsEnabled gate decided server-side, per-turn fresh); the prober doctrine gains the map-review nudge at session start. Also: runner's JobKind special-casing collapses (KIND_ROLE/KIND_RUNNERS are plain records again), semantic-clusters.ts dies with its test, the embedder dep's doc tells the wave-7 stub story, and ports.ts's readClusterLabels/readNamingRuns surface stays untouched for Platform's ledger-port interlude. 848/848 tests, typecheck/lint/build green. |
||
|
amosblomqvist
|
0fcffdffaf |
Birth-session clustering + the prober graph-tools data seam (wave 7 chunk A)
Embeddings are dying (chunk B removes the machinery); the clusters they
used to carry become the session that created them — derived on read.
- New src/core/session-clusters.ts: birthSessionPartition groups merged
nodes by their FIRST SURVIVING provenance point's chatId (owner-locked
2026-09-23). Deletion is self-healing by construction: stripped-birth
survivors re-attribute to their earliest remaining contributor, and
empty-provenance nodes stay unclustered. Notes imports are birth-sessions
too (owner ruling) — their label is the file path.
- graphSnapshot: clusters = the birth partition filtered >= MIN_CLUSTER_SIZE
(the hull presentation rule), id = session id, label = session name (chat
title / file path; a deleted record renders unlabeled). The persisted
cluster-labelings join dies; no LLM naming pass needed anymore.
- settleGraphLayout derives the same partition from readNodes — drops its
readEmbeddings dep. One cluster concept everywhere.
- Prober data seam (core/graph.ts, pure ports-driven like the snapshot):
graphSessionIndex (every birth session: id, name, chat|file kind,
createdAt/updatedAt, birth-proposition count, recency-ordered) and
graphSessionPropositions (a session's birth propositions with the
extractor's latest understanding level and THAT event's own timestamp —
owner ruling: quiz outcomes are ledger evidence, they never move the
evaluation date). Teacher's tool handlers consume these in-process.
- New agent.graphTools subsection { enabled?: boolean } (owner ruling:
the toggle gates prober tool REGISTRATION — agent capability, agent
section, its own subsection, NOT the tool-toggles map; absent = ON).
Lenient sanitizer, strict settings-body parser, passthrough resolution,
and graphToolsEnabled(effective) = graph.enabled && toggle for Teacher's
registry wiring.
Gates: tsc clean, eslint clean, 923/923 tests, next build compiles.
Live-verified on the worktree dev data: 10 nodes → 2 clusters labeled
with their chat titles; session index + propositions carry levels and
their own timestamps.
|
||
|
amosblomqvist
|
2678f04e7a |
feat: extraction coverage — message-granular ledger + token-batched runs (d132373 vision)
THE MESSAGE IS THE UNIT OF COVERAGE (vision-extraction-coverage.md): a per-chat append-only ledger (coverage.jsonl) records which messages the pipeline has extracted, by content fingerprint over exactly what the extraction view sees (one shared line builder — core/coverage.ts); the uncovered remainder is DERIVED at read time, never dual-tracked. A quiz answer merging into the asking message re-opens only that message; plan-block data-part attach never re-opens anything. - The trigger (turn-service) replaces every-3-quiz-answers with uncovered-tokens >= EXTRACTION.chunkTokens (4096) — quiz-less sessions extract too, content size is the honest signal. - runExtraction is now chunked: token-budgeted batches of the oldest uncovered consecutive messages (a tool call + result never split), each merged batch checkpointing its messages' coverage atomically — an abort or error loses at most the in-flight batch; merged batches stay covered. - Observation refs are batch-tagged (obs:<batchTag>:<sourceId>): re-runs of the same batch dedupe; a re-opened message's fresh understanding lands as a new event. Fixes the old obs:p1 position-collision. - New API: GET /api/graph/coverage (exact per-session derived coverage + running flags, most-uncovered first) and POST /api/graph/coverage/extract (manual run for one session, 409 while in flight). - Graph view chrome: the coverage button opens a panel with every session's covered/uncovered tokens and a per-session Extract button — the manual answer to aborted runs, errors, and dormant tails. Renders even when the graph is empty: an empty graph with uncovered sessions is exactly when it matters. Polls only while open. - Notes imports keep the whole-file checkpoint path (coverage undefined). - Tests: pure-core coverage suite (11), chunked-pipeline suite (3: multi-batch + no-op, abort-keeps-earlier-batches + remainder re-run, re-open alone + data-part invisibility), panel render suite (5); pendingQuizAnswers and its tests deleted as dead code. 486/486. Live-smoked on real GLM: manual extract of "teach me modal realism" ran one batch, wrote 4 coverage entries, panel flipped to "1.3k covered · complete" with the button disabled, graph refetched (nodes 11), usage persisted with real stats. |
||
|
amosblomqvist
|
f5edc7f0a5 |
Semantic topic clusters: local embeddings replace Louvain
Louvain only ever found the reasoning chains (the graph is disconnected by nature — every connected component became its own cluster) and edge-orphans could never join anything. Clusters are now RELATED TOPICS: propositions grouped by meaning. - Embedder (src/server/embeddings.ts): in-process transformers.js v4 running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download), MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs, L2-normalized so cosine = dot. Model/dim mismatch voids the ledger (full re-embed). Injected at the composition root — the agent layer never imports the server layer (architecture law). - Ledger (data/graph/embeddings.json): model + dim + vectors + the persisted cluster partition. Reclustering runs at the END of every embedding batch (post-merge hook, both front doors) — never per read (the agglomerative pass is ~1.5s at 338 nodes). - Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers stay singletons by default; attachStragglers is the all-in toggle. Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans land in real topics, 10 stragglers. - Consumers (snapshot, settle, namer, replay) read the persisted partition — one cluster concept everywhere; core-survival label matching unchanged (20/26 topics kept labels). - Louvain retired: detectCommunities + graphology* deps deleted (graphology-louvain had leaked into the client bundle since the clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed. - scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s including the one-time model download). |
||
|
amosblomqvist
|
0c14ea425d |
Option A: incremental anchored force layout, persisted server-side
The layout now EVOLVES instead of reshuffling on every extraction. A pure
core module (src/core/layout.ts) warm-starts forceAtlas2 from the positions
ledger (data/graph/positions.json, read-modify-write like sources.json,
updatedAt stamp for future live-physics conflict handling), re-blends every
previously-positioned node toward its anchor each iteration (retention 0.85
- old placements stay recognizable, strong forces still win), seeds new
nodes near their positioned neighbors, and seeds orphans on the rim outside
the core so they read as unconnected knowledge.
- Port/store: GraphStore.readPositions/writePositions; snapshot carries
positions when the ledger exists (optional - fresh graphs unchanged).
- Extraction pipeline settles synchronously after every merge (chat and
notes paths), before usage persist - the next /api/graph fetch already
reflects new nodes.
- View: the ledger IS the layout; the local one-shot FA2 pass survives
only as a fallback when no ledger exists.
- scripts/replay-layout.ts: simulates organic growth by replaying the
current 338-node graph one imported file at a time (real import order,
edges activate when both endpoints exist). Real run: 338 positions in
0.3s, 90-degree-0 orphans collecting at mean r=338 vs connected r=194.
- Also restores the cluster region fills (hull background colors) that
|
||
|
amosblomqvist
|
f0267a5fd9 |
Clusters phases B+C: labeling records, the naming call, snapshot payload
Core-survival labelings (vision section 1, DECIDED 2026-09-13): ClusterLabeling records in a new append-only ledger (data/graph/cluster-labels.jsonl), immutable, id = clu_ + sha256 of the sorted core so re-naming races and re-runs dedupe at the store seam. matchClusterLabelings (core/clusters.ts, pure): a community keeps a record's label while at least half its core survives in it; both sides single-successor (a split keeps one name; greedy by overlap, ties to the newer record). The namer (src/agent/subagents/cluster-namer.ts): ONE singular cheap LLM call (new 'labeler' role, glm-flash/low) names every new/changed community at once, given member names + statements (capped at 12 members per cluster). refreshClusterLabels runs POST-EXTRACTION (user-decided) from both front doors, fire-and-forget — naming failure never fails the run; refreshes serialize behind a chain so parallel imports never mint duplicate labels. import-notes.ts awaits clusterNamingSettled before exit. Payload: GraphSnapshot.clusters (>= MIN_CLUSTER_SIZE 3) joins detection with labelings at read time — claimed clusters carry the record's label and id; unlabeled ones carry a member-hash id and null label. Detection is deterministic so no memo in v1 (ms at current scale; revisit at vault scale). Tests (372/372): matcher half-rule/split/merge/greedy/tie cases; naming end-to-end through importNotes (one call, one record, claim on re-refresh), naming failure non-fatal; snapshot derivation with labeling join; 11 test pipelines now pin an explicit labeler seam. |
||
|
amosblomqvist
|
543cf33789 |
Notes front door: format-agnostic import into the graph pipeline
The same extractor→auditor→merger pipeline now runs over raw personal
notes in arbitrary formats (vision §1, DECIDED 2026-09-13):
- NotesSource records (data/graph/sources.json, deterministic id from
path hash → idempotent re-imports) play the chatId role in mappings,
checkpoints, and observation events; usage lands on the source record
(appendSourceUsage, idempotent per startedAt, kept across upserts).
- mergeFindings is chat-store-free: callers pass MergeSource
{messages, foldedAt} — sessions pass the chat record, notes pass
messages:null (no quiz-event rebuild; a deterministic rebuild needs
structured quiz tool parts).
- Second extractor prompt (NOTES_EXTRACTOR_INSTRUCTIONS) for general-
purpose extraction; understanding judges the AUTHOR's grasp. Auditor
wording input-agnostic. extract/audit share one generic runSubmitAgent.
- graphSnapshot.fileSources (srcId→filePath); notes sources render as
static path entries in the detail card (FileTextIcon, never a link).
- One shared pipeline instance (exported from chat-handler): same
single-flight registry, live status chips (label = file path), usage
accounting, and true abort for both front doors.
- Transport: POST /api/graph/import (multipart, 3-wide worker pool) +
Import notes… button in the graph view; scripts/import-notes.ts for
batch runs (concurrency 4).
- real-data-extraction/ gitignored (raw import sources never committed).
355/355 tests; typecheck, lint, build green.
|
||
|
amosblomqvist
|
5653c77a29 |
feat: probe/teach agent split — explicit phase markers, position-only plan map
Diagnosis (2026-09-11 user testing, minimal-effort GLM): sessions stuck on 'plan' forever. Two causes: (1) the three-phase view read the stored tree's currentNodeId, which nothing ever writes post author-once (2026-09-03) — a structural dead seam; (2) the teacher stochastically left quiz.nodeId untagged, and done-by-quiz-result semantics would have falsely completed nodes anyway. The redesign (docs/probe-teach-split.md) abolishes derived occupancy in favor of explicit events: - Two agents over one transcript: createProberAgent (quiz without a nodeId field, ask, probe_done) and createTeacherAgent (full toolset, plan_init onward). The chat-handler construction branch keys on a probe_done part in the transcript. MODELS.prober added. - probe_done: server-side seam tool — carries the learner assessment (strands + free-form edge read), refuses while a question is open or the probe is already concluded, and queues a handoff steer through the generalized wake pipeline (steers now dedupe by id, not jobId; job finalization only for job steers). The teacher auto-wakes and authors the plan; the learner sees the assessment via the probe-done card. - phases: probe ends at a concluded probe_done, plan ends at a successful plan_init — strictly sequential, no skipping, no legacy compatibility (data wiped at the pivot per doctrine). - Plan map = position, not completion: derivePlanStatuses reads only tagged quiz CALLS (state/outcome feed nothing); current = latest tagged node with consecutive duplicates collapsed, visited = earlier distinct stays (a revisit stays in both — current wins at render). PlanTree is structure-only; currentNodeId/doneNodeIds and the dead mutation helpers are gone. No fallback current. - Prompts split per agent (prompts/prober/, prompts/teacher/ over the trimmed shared base); graph-context injection removed entirely (shelved, docs/probe-teach-split.md §5); knowledge-graph context no longer reaches either agent. - Plan blocks are full-only snapshots at plan_init (neighborhood machinery retired; the persistent DAG window is the future UI wave); usage capture unchanged. Gates: tsc clean, 294/294 tests (phases/plan-tree/plan-blocks/question- tools rewritten to the new semantics; probe-done + handoff-wake tests added), lint, build. Live GLM smoke PASS: prober concluded with an honest assessment, teacher auto-woke, authored a 10-node plan with the block attached, then tagged checks on slope-instinct → secant derived current/ visited exactly as designed; grading honest both ways (driver off-by-one proved a wrong answer grades false). |
||
|
amosblomqvist
|
9174cde55e |
Extraction observability: usage capture, live status payload, true abort (Wave 4)
Extraction is real spend and background work; the dock needs to see and stop it. User decisions: persist all cost data (chat record), per-stage usage granularity, and TRUE cancellation for abort. USAGE CAPTURE (live + persisted): - extraction.ts: per-call onStepFinish captures each billed step's usage (SDK LanguageModelUsage -> CallUsage: input/output/cache read+write/reasoning) into the registry's live per-stage usage (extractor/auditor; the merger is deterministic — no model call). - /api/graph/status runs now carry usage (model + steps per stage); cost/context folds are the consumer's (core/prices.ts — nothing stores dollars). - ChatRecord.extractions: append-only per-run usage log (mirrors the turns log): startedAt key, at, durationMs, merge stats when the merger ran, stages. Persisted on success AND failure — a failed run still spent money. Idempotent per startedAt; the list payload (ChatSummary) never leaks it. - EmissionStats moved to contracts/graph.ts (its contract home); core/graph.ts re-exports for existing importers. - Verified end-to-end against the real pipeline (backfill one chat): per-step token counts, reasoning tokens, and the auditor's growing prompt-cache reads captured and persisted. TRUE ABORT (the dock's abort button, user decision): - RunningExtraction owns one AbortController per run, shared across phases; abortExtraction(chatId) fires it and returns whether a run was aborted (idempotent no-op otherwise). The pending model call receives the signal — the provider cancels the request, spend stops. throwIfAborted() at phase boundaries ends runs aborted in sync gaps. - Abort is a deliberate stop, not a failure: same ledger discipline (nothing merged, nothing checkpointed, partial usage persisted) but the run reports zero emissions instead of throwing. Exported via src/server/observability.ts for master's /api/activity/abort route. - Tests: mid-run abort (hanging mock that rejects on the signal), live-accumulation + persisted-record assertions, store-seam idempotency, and the summary-payload leak guard. |
||
|
amosblomqvist
|
0fccbbe2ad |
Detail card: per-session understanding notes next to the session list
The extractor's per-proposition understanding observations (named-word scale: solid/shaky/accepted/...) were ledger data since the redesign, with no surface. This exposes them at read time and shows each session's note as a small chip on its session row in the node detail card — the per-session confidence history the user asked for, one observation per (node, chat), latest wins within a chat. - GraphSnapshot.nodeObservations: nodeId -> chatId -> note, built in graphSnapshot() from observation events (pure read-time join). - Sessions list rows: link + observation chip. - Session-relative judgments stay session-scoped: they render next to their session, never as node state (the state model stays pending). |
||
|
amosblomqvist
|
54f88f37e1 |
Strip the transitional node-state classes (2026-09-08)
The five-class NodeStateClass (taught/known/missed/dont-know/candidate) was transitional scaffelage from the 2026-09-02 redesign, deriving state by latest-quiz-wins — exactly what the philosophy rejects: understanding is inferred at read time from the real state model (still pending ideation), never encoded as quiz-derived classes. - contracts: NodeStateClass deleted; GraphSnapshot no longer carries nodeStates (objective graph + sessions only). - core: deriveNodeStates replaced by latestQuizOutcomes — the raw event fact (nodeId -> latest quiz kind), consumed ONLY by the teacher's graph-as-context selection, never by the home renderer. - graph-context: hotspot selection re-keyed to outcomes — groups are now 'Hotspots (last answer wrong or don-t-know) / Answered correctly / Never probed'. Same diagnostic selection, no state taxonomy. - graph-view: nodes render uniformly (accent color, size 8); the panel is search-only; the detail card's state badge is gone. Quiz outcomes remain ledger data for the future state model. Verified live (headless): uniform discs (single color/size, zero state pixels), search-only panel, card = name + statement + sessions, no console errors. |
||
|
amosblomqvist
|
bdd9cac2b7 |
Node schema pivot: short name (label) + full-precision statement; description dropped
With real extracted data, full-sentence statements as canvas labels proved unworkable: 10-line label blocks, illegible overprint clusters, bottom-edge clipping. Per the user's 2026-09-08 decision: - GraphNode/ExtractedProposition gain `name` — a very short, glanceable label (glossary-headword style: "Will to power"), presentational only: not unique, not identity, never precise. Set at creation; link-appends never change it (first line defines; labels stay stable across sessions). - `statement` keeps its full precision and becomes the identity anchor; the detail card shows it under the name. `description` is dropped. - Extractor coins the name (freshest session context); auditor carries it through on create, rewording only clearly unglanceable names; linked propositions keep the existing node's name. - Renderer: canvas label = name; search matches name + statement; detail card = name (bold) + statement (muted). - Store guard requires name (pre-pivot node lines are skipped as legacy); vision.md node-schema bullet + amendments table updated (2026-09-08). |
||
|
amosblomqvist
|
b07c907ffd |
Graph redesign: true propositions, follows/supports, auditor-owned identity (vision §1 minimal implementation)
The minimal implementation of the 2026-09-02 graph vision (docs/vision.md §1):
- contracts/graph.ts: every node is a TRUE proposition (statement is both
rendering label and identity key; required description; no kind, no name).
Edges are follows (derivable) / supports (grounds without entailing) with
message-granular provenance points {chatId, messageId}. Events reduce to
correct/miss/dont-know plus observation (understanding notes); no taught
events, no misconception kind.
- extraction.ts: extractor is TREE-BLIND (transcript only, no plan tree) and
free-form; auditor sees findings + the current graph, owns faithfulness and
identity (link-or-create verdict per proposition, recorded as linkTo);
transcriptView tags every line with its message id (provenance anchors).
- core/graph.ts merger: applies verdicts deterministically — linkTo appends
a link line (phrasings join aliases, provenance unions; reads merge by id),
absent creates; NO string matching anywhere. Quiz events rebuilt from tool
parts with attribution-only nodeId resolution; understanding notes become
observation events (obs:<sourceId>); mapping lines are the verdict audit
trail and make re-merges idempotent.
- deriveNodeStates: transitional — latest quiz event wins (known/missed/
dont-know, else candidate); taught unreachable until the state-model
ideation lands.
- graph-context.ts + 50-graph-context.md: follows chains, hotspot grouping,
state vocabulary updated for the new event kinds.
- graph-view.tsx: node label is the statement; detail shows statement +
description.
- tests: merger verdicts (link, no-string-match, cross-chat assertions),
tree-blind pipeline end-to-end, follows vocabulary in context tests.
276/276 green.
|
||
|
Amos
|
2e4e43b8c4 |
Graph nodes gain a name; nodes are objective facts (no learner/session references)
Schema: GraphNode + ExtractedProposition carry 'name' — a short 2-6 word header, the rendering handle. Node theory made explicit: a node is an objective world-facing fact; the learner's relationship to it lives entirely in the event ledger (state labels are derived, never stored). Extractor + auditor instructions enforce it — misconceptions state the wrong idea itself as a claim, never 'the learner believes Y'. Legacy ledger lines without a name derive one from the statement on read. graph-view: labels use the name; the detail panel shows name as header and statement as body. |
||
|
Amos
|
af90fadcda |
Extraction pipeline: extractor → auditor → merger — the sole graph writer (stage 5)
The knowledge graph's write channel is now the pivot's target
architecture (vision §1 construction):
- src/agent/subagents/extraction.ts — the pipeline factory
(createExtractionPipeline({ chatStore, graphStore, extractor?,
auditor? }); models env LH_EXTRACT_MODEL ?? LH_MODEL):
* transcriptView(): the compact view both LLM stages see — final plan
tree with node ids, conversation verbatim, QUIZ lines with outcomes,
mark_done lines; job tool parts are one-liners, plan ops fold into
the tree line.
* Extractor: transcript → SessionFindings (propositions incl.
misconceptions and prose-only supports edges, attribution for
UNTAGGED probes, all with verbatim evidence) via one zod-validated
submit_findings call.
* Auditor: transcript + findings → the verified subset + drops (the
faithfulness net) via submit_review.
- src/core/graph.ts — emitChat/foldStaleChats/replayFinalTree retire;
mergeFindings() replaces them as the pipeline's deterministic apply
stage: exact/alias identity resolution (one mapping line per (chat,
graph node)), edge dedupe, and event rebuild FROM THE TRANSCRIPT's tool
parts — kinds/timestamps/refs/notes deterministic, the LLM's only event
say being WHICH proposition an untagged probe tested (retroactive —
the vision's open question #3, resolved structurally: tagged events
resolve deterministically; attribution rescues untagged and
tagged-but-unmapped ones; unattributable land null + question text).
pendingQuizAnswers() is the trigger helper.
- Trigger (turn-service.ts triggerExtraction, replacing the old
foldTurnIntoGraph body): once a chat accumulates
EXTRACTION.everyNQuizAnswers (LH_EXTRACT_N, default 3) new answered
quizzes, fire the pipeline async — single-flight per chat, failure
checkpoints nothing and re-runs on the next trigger. Deliberately a v1
placeholder isolated in ONE seam, trigger-agnostic pipeline.
- GET /api/graph is now a pure read (foldStaleChats gone).
- NodeMapping.treeNodeId → sourceId (the mapping key is now the writer's
local proposition id — tree node when tree-derived); store strict-read
schema updated. Local ledger data regenerates through the pipeline.
- Tests: graph.test.ts rewritten around mergeFindings (identity,
attribution paths incl. ghost-rescue, idempotency, cross-session
provenance, trigger counting), graph-context.test.ts merge section
converted to findings-driven, NEW extraction.test.ts — transcriptView
units + the full pipeline with mock models (audit drops, failure
checkpoints nothing, single-flight). 195/195, tsc/eslint/build clean.
Probe attribution (vision open question): probes attach retroactively by
the extractor against the whole transcript; a pre-tree probe maps to a
proposition the tree only articulates later. Fuzzy/embedding merge
adjudication stays deferred (conservative exact/alias only).
|
||
|
Amos
|
1167157c23 |
Storage ports: ChatStore/GraphStore/JobStore + adapters + core inversion (stage 3)
Per-entity semantic ports (src/core/ports.ts), one adapter family
(src/store/*, fs JSON/JSONL), exactly one composition point
(src/server/stores.ts). Dependency direction is now machine-checked:
adapters implement core interfaces; core never imports an adapter.
The inversion:
- core/graph.ts (emitChat/getGraph/graphSnapshot/foldStaleChats) is now
pure — it takes the storage ports as explicit parameters (GraphDeps).
No store imports; newGraphNodeId is local; node id format unchanged.
- core/graph-context.ts buildGraphContext() no longer defaults to the
live ledgers: the chat-handler hands in getGraph(graphStore) per turn.
- store adapters export exactly one object each implementing their port
(chatStore/graphStore/jobStore); bodies otherwise verbatim.
NodeMapping + SessionCheckpoint moved store/graph -> core/contracts
(they ARE contract shapes); CreateJobInit/JobPatch live in ports.
- titleFromPrompt moved store/chats -> core/contracts (pure helper,
planNodeStatus precedent). Store modules now contain persistence only.
- server/stores.ts is the composition root; server/{chats,graph,jobs}.ts
and chat-handler wire from it; agent (plan tools, sub-agent runner)
call the port objects. The shell surface (@/src/server exports) is
unchanged.
Lint law extended: the whole of src/core is now a purity zone (was a
file glob with a graph/graph-context exemption). Verified it fires on
core->store imports; tsc clean, 190/190 tests, eslint clean, prod build
clean (prerender 6/6). Tests pass deps explicitly — graph.test uses the
same fs-adapter pair stores.ts wires in production; graph-context.test
keeps its in-memory ledgers as a GraphStore implementation.
|
||
|
Amos
|
e92140f993 |
Shell/core import law: public API barrels + ESLint enforcement (stage 2)
The shell now imports through exactly two barrels, enforced by
import/no-restricted-paths (npm run lint):
components/** -> "@/src/core" (pure core API, browser-safe)
app/** -> "@/src/core" + "@/src/server" (server entries)
Zone violations are lint errors with architecture-pointer messages; the
pure-core modules (contracts, question-core, plan-tree, phases, pending,
math-delims, ui-message) may never import agent/server/store.
Supporting moves (direction: IO layers -> core):
- core/index.ts: the core public API barrel
- core/ui-message.ts: moved from src/agent (it was shell-facing all along)
- quiz/ask input zod schemas moved agent/tools -> core/question-core
(the question contract, alongside the grading logic)
- ChatRecord/ChatSummary moved store/chats -> core/contracts (single schema)
- GraphSnapshot/NodeStateClass moved core/graph -> core/contracts (the
/api/graph payload IS a contract; the pure renderer needs only these)
- src/server/{chats,graph,jobs}.ts + index.ts: thin server entries for the
transport layer; GET /api/graph's fold side-effect now lives behind
graphView() instead of inline in the route
Gates: tsc clean, 190/190 tests, eslint clean (law verified to fire on
store/agent/subpath violations), production build clean.
(Repo repair en route: a git object (app/chat/[id] tree) was missing from
the object store — reconstructed byte-exactly via mktree from its blob;
the broken node_modules that stalled the build was cleanly reinstalled.)
|
||
|
Amos
|
806e1405d5 | W8: graph-as-context injection + exact-match merge | ||
|
Amos
|
8728573253 | W4: graph store + emission + home graph view |