sorokin.oleg.edu/learnharness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

extraction: the spine stage — cross-stretch assembly between extract and audit

Browse Source 
Per-batch pipeline is now extract → spine → audit → merge (owner order,
2026-09-30), the spine inside the serialized audit section so it reads
the graph exactly as the following audit will. Measured motive: 0 of 133
chains in the live corpus crossed a stretch boundary — no stage minted
cross-stretch chains.

- src/agent/subagents/spine.ts: SESSION/NOTES spine preambles + shared
  SPINE_CORE (span doctrine; extraction doctrines carried over), the
  submit_spine TypeBox schema, validateSpine (mechanical span law: every
  chain touches >=1 fresh endpoint AND >=1 existing digest endpoint;
  unknown ids, self-reference, and id collisions are loud), the digest
  prompt builder, and the runner (strong tier, retry-only).
- extraction.ts: 'assembling' phase, spine stage in the per-need gate,
  usage/billing stage, assemble() inside withAuditLock for both front
  doors; first stretch of a source skips the call (empty digest, no
  spend); the audit sees COMBINED propositions (chunk + connectives),
  never chains; merger receives the combined findings.
- core/graph.ts mergeFindings: chain endpoints resolve through the
  job-local map OR pass through verbatim as existing n_ ids (the spine's
  mixed namespace).
- EXTRACTION_CORE: steps-of-reasoning selection bar (examples, notation
  glosses, and restatements that no argument needs are not steps).
- Factory chunk budget 4096 -> 8192 (fewer seams, more local context).
- healStringifiedArrays moved to the shared submit seam (submit-agent.ts).
- Cost panel: Assembly row; usage records carry a spine stage (optional
  on legacy records).
- Tests +5: prompt pins, span-law matrix, digest rendering, seeded
  end-to-end flow (spanning chain + connective, verbatim n_ endpoint,
  spine billed as its own stage), non-spanning refused + retried.
This commit is contained in:
amosblomqvist
2026-09-30 16:38:45 +03:00
parent 5ed724c224
commit 52c02533d1
18 changed files with 787 additions and 87 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+6 -4
View File
@@ -239,7 +239,7 @@ flowchart TB
end
subgraph LIBRARY["LIBRARY — world model (owns data/graph ledgers)"]
EXTRACT["extraction pipeline — extractor → auditor → merger (SOLE graph writer)"]
EXTRACT["extraction pipeline — extractor → spine → auditor → merger (SOLE graph writer)"]
DERIVE["derivation — graphSnapshot · birth-session clusters (reads, never write)"]
GRAPHDATA["append-only ledgers — data/graph/*.jsonl"]
end
@@ -339,7 +339,7 @@ Sub-agents **verify, never one-shot**: the demo-builder headlessly executes its
```mermaid
flowchart TB
TRIG["trigger seam (turn-service): coverageOf(record, coverageLedger)<br/>— uncovered tokens ≥ graph.extractionChunkTokens (settings)?"] -->|"yes, single-flight per source"| RUN["extraction.runExtraction(chatId)"]
IMPORT["POST /api/graph/import (notes front door)<br/>deterministic src id = sha256(path); content-fp guard skips unchanged re-imports"] -->|"importNotes: shared pipeline, second extractor prompt"| RUN
IMPORT["POST /api/graph/import (notes front door)<br/>deterministic src id = sha256(path); content-fp guard skips unchanged re-imports"] -->|"importNotes: shared pipeline, per-door preambles"| RUN
RUN --> BATCH["batchRuns — greedy token-budgeted batches of oldest uncovered TEACHER-PROSE messages"]
EXT["EXTRACTOR (LLM) — sees ONLY the teacher-prose view (tree-blind, graph-blind)<br/>submits propositions + REASONING CHAINS (owner redesign, 2026-09-30)<br/>stored AS-IS — chains.jsonl is the truth (chains-primary, 2026-09-30)"]
EXT --> AUD["AUDITOR (LLM, agentic loop) — sees findings ONLY, never the source;<br/>explores the graph via find / list_sessions / list_propositions tools,<br/>judges identity ONLY (link-or-create, equivalence bias) — sees propositions, never mints chains"]
@@ -350,11 +350,13 @@ flowchart TB
**Raw reasoning chains only** (2026-09-27 cleanup; teacher-prose view 2026-09-28): the extractor reads ONLY the session — and since the 2026-09-28 ruling, ONLY the teacher agent's prose: plain paragraphs under the session title, no learner messages, no thinking, no tool calls or results, no quiz blocks. Coverage fingerprints, slot tokens, and batch budgets all derive from that same selection (`messageViewLines` in `core/coverage.ts`), so "covered" is exactly "what the extractor saw" — a quiz answer or learner edit can never open a run, and only a teacher-prose edit re-opens its message. The old call+result inseparability bond is deleted (nothing left to keep together); batching is pure greedy fill. The shared markdown engine (`core/session-markdown.ts`) keeps the full session format for the future user-facing export and is no longer used by extraction. The extractor's deliverable is propositions + REASONING CHAINS (owner redesign, 2026-09-30): a chain is premises that JOINTLY entail a conclusion — independent arguments for the same conclusion are separate chains, conclusions are reusable by id as premises, and a bare proposition rides the propositions list alone. The wire schema carries propositions (id/name/statement/optional aliases) + chains (premises minItems 1/conclusion) — and since the chains-primary ruling (2026-09-30) the STORE carries exactly that wire shape: chains append as-is to `chains.jsonl` (`SessionFindings` IS the store shape now; `chainsToEdges` and `chainsToGraph` are deleted — no translation seam exists). **Supports edges are retired entirely** (owner, same evening): `EdgeType`, the supports draw toggle and thickness slider, and every supports code path are gone — a stored typed edge no longer exists anywhere, and every arrow on the canvas is derived from a chain. No migration, no backcompat: the pre-chains graph data was wiped by ruling. The chain laws are enforced by the practice-proven error-back hook at the submit seam: every premise/conclusion id must resolve to a submitted proposition, premises are non-empty, no self-reference — a violation returns an error string and the model re-submits within its turns. The prompt is GOAL-FIRST, no rule armor (owner philosophy ruling, 2026-09-30): one shared `EXTRACTION_CORE` (the goal, the truth law, the submission mechanics) plus two one-sentence contextualizations — session and notes preambles — assembled at the call sites, so both front doors share the core by construction; the old defensive lists (DROP lists, merge-your-own-output, name walls, when-in-doubt-omit, the faithfulness paragraph) are all cut, and the old opening describing Quiz/Recall-check/Question blocks (stale since the teacher-prose ruling) is simply omitted. A `prepareArguments` heal at the pre-validation seam rescues stringified-payload submits (propositions or chains). Every node is an objective, world-facing fact. No quiz outcomes, no understanding assessments — the events ledger and `UnderstandingLevel` are deleted; the learner's relationship to a proposition is practice territory — which landed 2026-09-29 as **feedback-only strengthening** (§7): grades live on problem sets, and nothing ever writes learner states back to the graph.
**The spine — cross-stretch assembly** (owner-approved stage, 2026-09-30): extraction walks a source in stretch-blind batches, and the live corpus showed the seam — 0 of 133 chains crossed a stretch boundary, because no stage minted cross-stretch chains (the extractor sees one batch, the auditor links propositions only, the merger is deterministic). The spine closes it: after each batch's extract and INSIDE the same serialized audit section (owner-specified order — extract → spine → audit → merge, so the spine reads the graph exactly as the following audit will), one strong-tier call sees the same-source digest (every merged proposition of this source, ids verbatim, plus the chains already recorded) and the fresh stretch's findings, and submits only what SPANS them: cross-stretch chains and the connective propositions a chain needs that no stretch stated on its own. The span law is mechanical, enforced at the submit seam (error-back like every chain law): every chain must touch ≥1 fresh endpoint and ≥1 existing digest endpoint — all-fresh is the stretch extractor's own work, all-existing already had its chance when the later endpoint arrived. Identity stays the auditor's lane: the spine submits no links; its connective propositions ride into the audit as ordinary findings (propositions only — the auditor still never sees chains), and its chains resolve through the merger's job-local map, which now accepts existing graph ids verbatim as endpoints (the mixed namespace: fresh `p`/`s` local ids OR digest `n_` ids). Session and notes front doors share the stage exactly as they share `EXTRACTION_CORE` — only the preamble differs. The first stretch of a source skips the call entirely (empty digest — nothing to join, no spend). The spine bills as its own usage stage (`spine` in the run record and the cost panel's Assembly row) and runs under its own phase (`assembling`).
**Serialized audit, parallel extracts — per-chunk jobs** (owner ruling 2026-09-28): every chunk (notes file) and every batch (chat coverage) is its own JOB — registry row, usage record, failure domain. Extracts fan out in parallel across all jobs and sources, throttled by a process-wide 3-permit extract semaphore on `globalThis` (the provider-facing rate stance; a job waiting for a permit shows `queued`). The audit→merge section remains a process-wide FIFO critical section — a mutex on `globalThis` (the same module-instance idiom as the run registries): before any auditor LLM call or graph read, a job waits (`waiting` phase) until every earlier job's merge has landed, so each auditor explores the graph as it actually stands — two concurrent audits would each link into a graph the other is about to change, the structural mint of duplicate nodes. Failure isolation: one job failing (bad chunk, degraded model output) is absorbed at the fan-out — its messages stay uncovered and honestly re-runnable while its siblings still merge; the run returns what actually merged. Abort is source-wide: every job of a source's fan-out dies together, and merged jobs keep their checkpoints. The status feed shows one row per job (`file.md · 2/5` suffixes for chunked sources), and each job persists its own usage record (failed jobs bill their partial spend too).
**Division of trust** (the pipeline's core rule): the extractor is fully trusted and is the only stage that ever sees the source. The auditor never sees the transcript — it receives the findings' PROPOSITIONS ONLY (the chains never reach the auditor: they are the extractor→store deliverable and the merger's business — owner ruling, 2026-09-30) and explores the existing graph through read-only tools (`find`, `list_sessions`, `list_propositions` — statement text included, no timestamps) to reach its verdicts. VERDICTS-ONLY (owner ruling 2026-09-28): the auditor's submit is `{ links }` — identity links only (which findings propositions ARE existing nodes). The edges argument is GONE (owner, same evening): the auditor never creates relations between propositions — cross-source connection happens solely through identified equivalences, and argument structure comes from the extractor alone. It never restates the findings; the merger joins findings + verdicts, so an order of magnitude less is emitted and there is no structural way for the auditor to alter or drop extractor output. Equivalence bias (owner ruling 2026-09-27): paraphrases, rewordings, and compressions of the same claim MERGE; only genuinely distinct propositions stay apart. The deterministic merger validates and appends: link resolution, chain resolution through the job-local map, chain dedupe by (sorted premises, conclusion) — a known argument re-asserted by another chat appends under the SAME `arg_` id (link-style: the first line's id and createdAt stand, provenance unions) — coverage checkpoints. There is no drop channel — nothing rejects the extractor's output. **Per-job in-memory resolution** (owner, 2026-09-29): the mappings ledger is deleted — each job resolves its own proposition ids against the graph in memory and merges independently, so no job can see or collide with another's ids (the persisted table once silently dropped same-id propositions across chunks and welded wrong edges). Cross-job and cross-run identity is the auditor's links alone, plus the coverage ledger (nothing re-extracts) and the notes content-fingerprint guard.
**Division of trust** (the pipeline's core rule): the extractor is fully trusted and is the only stage that ever sees the source. The spine sees the graph digest + the fresh findings — never the source — and mints structure only (spanning chains + connectives; no identity verdicts). The auditor never sees the transcript — it receives the findings' PROPOSITIONS ONLY (the chunk's propositions plus the spine's connectives; the chains never reach the auditor: they are the extractor→spine→store deliverable and the merger's business — owner ruling, 2026-09-30) and explores the existing graph through read-only tools (`find`, `list_sessions`, `list_propositions` — statement text included, no timestamps) to reach its verdicts. VERDICTS-ONLY (owner ruling 2026-09-28): the auditor's submit is `{ links }` — identity links only (which findings propositions ARE existing nodes). The edges argument is GONE (owner, same evening): the auditor never creates relations between propositions — cross-source connection happens solely through identified equivalences, and argument structure comes from the extractor and the spine alone. It never restates the findings; the merger joins findings + verdicts, so an order of magnitude less is emitted and there is no structural way for the auditor to alter or drop extractor output. Equivalence bias (owner ruling 2026-09-27): paraphrases, rewordings, and compressions of the same claim MERGE; only genuinely distinct propositions stay apart. The deterministic merger validates and appends: link resolution, chain resolution through the job-local map (which passes existing `n_` ids through verbatim for the spine's mixed-namespace chains), chain dedupe by (sorted premises, conclusion) — a known argument re-asserted by another chat appends under the SAME `arg_` id (link-style: the first line's id and createdAt stand, provenance unions) — coverage checkpoints. There is no drop channel — nothing rejects the extractor's output. **Per-job in-memory resolution** (owner, 2026-09-29): the mappings ledger is deleted — each job resolves its own proposition ids against the graph in memory and merges independently, so no job can see or collide with another's ids (the persisted table once silently dropped same-id propositions across chunks and welded wrong edges). Cross-job and cross-run identity is the auditor's links alone, plus the coverage ledger (nothing re-extracts) and the notes content-fingerprint guard.
**Coverage, not quiz-counting** (supersedes the v1 every-N-quiz-answers trigger): each message's *view* (the compact line format `core/coverage.ts` has always hashed) is hashed into a per-chat fingerprint ledger (`coverage.jsonl`); "covered" means exactly "a run committed on this content." The prompt view is the richer markdown document — a deliberate 2026-09-27 split: re-formatting presentation never re-opens covered history. Uncovered = fingerprint diff, never dual-tracked; runs are greedy token-budgeted batches (`graph.extractionChunkTokens` — a settings knob, default 4096, floor 500, read fresh each turn) of the oldest uncovered messages — quiz-less sessions extract too. A coarse `sessions.jsonl` checkpoint still marks each merged run. The trigger seam (`turn-service.ts`) is trigger-agnostic from the pipeline's side: when a better mechanism lands (session end, idle, explicit flush), only the seam changes. Single-flight per source; the run registry powers the coverage UI and true abort.
**Coverage, not quiz-counting** (supersedes the v1 every-N-quiz-answers trigger): each message's *view* (the compact line format `core/coverage.ts` has always hashed) is hashed into a per-chat fingerprint ledger (`coverage.jsonl`); "covered" means exactly "a run committed on this content." The prompt view is the richer markdown document — a deliberate 2026-09-27 split: re-formatting presentation never re-opens covered history. Uncovered = fingerprint diff, never dual-tracked; runs are greedy token-budgeted batches (`graph.extractionChunkTokens` — a settings knob, default 8192 since the spine stage doubled it 2026-09-30, floor 500, read fresh each turn) of the oldest uncovered messages — quiz-less sessions extract too. A coarse `sessions.jsonl` checkpoint still marks each merged run. The trigger seam (`turn-service.ts`) is trigger-agnostic from the pipeline's side: when a better mechanism lands (session end, idle, explicit flush), only the seam changes. Single-flight per source; the run registry powers the coverage UI and true abort.
**Two front doors, one pipeline**: chat transcripts (quiz-triggered) and raw personal notes (`POST /api/graph/import` — each file becomes a `NotesSource` playing the `chatId` role, deterministic id from the path hash; a content-fingerprint stamp makes an unchanged re-import a TRUE no-op — no model calls, no usage — while changed text re-runs the whole file, owner 2026-09-29). Notes chunk like chats (owner, 2026-09-28): large files split at line boundaries under the same token budget, and each chunk/batch fans out as its own job exactly like a chat run — parallel extracts under the semaphore, serialized audits under the gate. No single model call carries a whole file. Both submit tools require their top-level fields (`chains`/`links` present even when empty): a payload missing them fails validation rather than silently merging structure-less nodes.