sorokin.oleg.edu/learnharness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Extraction pipeline: extractor → auditor → merger — the sole graph writer (stage 5)

Browse Source 
The knowledge graph's write channel is now the pivot's target
architecture (vision §1 construction):

- src/agent/subagents/extraction.ts — the pipeline factory
  (createExtractionPipeline({ chatStore, graphStore, extractor?,
  auditor? }); models env LH_EXTRACT_MODEL ?? LH_MODEL):
  * transcriptView(): the compact view both LLM stages see — final plan
    tree with node ids, conversation verbatim, QUIZ lines with outcomes,
    mark_done lines; job tool parts are one-liners, plan ops fold into
    the tree line.
  * Extractor: transcript → SessionFindings (propositions incl.
    misconceptions and prose-only supports edges, attribution for
    UNTAGGED probes, all with verbatim evidence) via one zod-validated
    submit_findings call.
  * Auditor: transcript + findings → the verified subset + drops (the
    faithfulness net) via submit_review.
- src/core/graph.ts — emitChat/foldStaleChats/replayFinalTree retire;
  mergeFindings() replaces them as the pipeline's deterministic apply
  stage: exact/alias identity resolution (one mapping line per (chat,
  graph node)), edge dedupe, and event rebuild FROM THE TRANSCRIPT's tool
  parts — kinds/timestamps/refs/notes deterministic, the LLM's only event
  say being WHICH proposition an untagged probe tested (retroactive —
  the vision's open question #3, resolved structurally: tagged events
  resolve deterministically; attribution rescues untagged and
  tagged-but-unmapped ones; unattributable land null + question text).
  pendingQuizAnswers() is the trigger helper.
- Trigger (turn-service.ts triggerExtraction, replacing the old
  foldTurnIntoGraph body): once a chat accumulates
  EXTRACTION.everyNQuizAnswers (LH_EXTRACT_N, default 3) new answered
  quizzes, fire the pipeline async — single-flight per chat, failure
  checkpoints nothing and re-runs on the next trigger. Deliberately a v1
  placeholder isolated in ONE seam, trigger-agnostic pipeline.
- GET /api/graph is now a pure read (foldStaleChats gone).
- NodeMapping.treeNodeId → sourceId (the mapping key is now the writer's
  local proposition id — tree node when tree-derived); store strict-read
  schema updated. Local ledger data regenerates through the pipeline.
- Tests: graph.test.ts rewritten around mergeFindings (identity,
  attribution paths incl. ghost-rescue, idempotency, cross-session
  provenance, trigger counting), graph-context.test.ts merge section
  converted to findings-driven, NEW extraction.test.ts — transcriptView
  units + the full pipeline with mock models (audit drops, failure
  checkpoints nothing, single-flight). 195/195, tsc/eslint/build clean.

Probe attribution (vision open question): probes attach retroactively by
the extractor against the whole transcript; a pre-tree probe maps to a
proposition the tree only articulates later. Fuzzy/embedding merge
adjudication stays deferred (conservative exact/alias only).
This commit is contained in:
Amos
2026-08-31 22:36:41 +03:00
parent 59e1fee850
commit af90fadcda
12 changed files with 1168 additions and 320 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/graph.test.ts
+125 -87
View File
@@ -1,21 +1,27 @@
/**
* Graph store + emission tests (W4; vision §1, contracts.ts "v1 emission").
* Graph merger + trigger + derivation tests (vision §1; contracts.ts
* "extraction pipeline"). The merger (core/graph.ts mergeFindings) is the
* pipeline's deterministic apply stage: findings in, ledger appends out —
* identity resolution, retroactive probe attribution, event rebuild, and
* the idempotency guarantees.
*
* Ledger isolation: PATHS is fixed at import time, so the five data/graph
* ledgers are backed up and cleared before the suite and restored after
* (try/finally-shaped: the after() hook always runs, even on failure).
* The chat fixtures are real ChatRecords in the chat store (created and
* deleted by this suite) — emission reads exactly what production reads.
* deleted by this suite) — the merger reads exactly what production reads.
* The full LLM pipeline (extractor → auditor) is tested in
* tests/extraction.test.ts with mock models.
*/
import assert from "node:assert/strict";
import * as fs from "node:fs";
import { after, describe, it } from "node:test";
import type { UIMessage } from "ai";
import type { PlanTree, PlanToolResult } from "../src/core/contracts.ts";
import type { PlanTree, PlanToolResult, SessionFindings } from "../src/core/contracts.ts";
import type { QuestionAnswerData } from "../src/core/question-core.ts";
import { createTree, markDone, removeNode } from "../src/core/plan-tree.ts";
import { emitChat, foldStaleChats, getGraph, graphSnapshot, type GraphDeps } from "../src/core/graph.ts";
import { createTree, markDone } from "../src/core/plan-tree.ts";
import { getGraph, graphSnapshot, mergeFindings, pendingQuizAnswers, type GraphDeps } from "../src/core/graph.ts";
import { graphStore } from "../src/store/graph.ts";
import { chatStore } from "../src/store/chats.ts";
import { LEDGER_FILES, PATHS } from "../config.ts";
@@ -32,7 +38,7 @@ const createdChats: string[] = [];
fs.mkdirSync(PATHS.graph, { recursive: true });
for (const file of LEDGER_PATHS) {
if (fs.existsSync(file)) {
const bak = `${file}.w4test.bak`;
const bak = `${file}.w5test.bak`;
fs.renameSync(file, bak);
backups.push({ file, bak });
}
@@ -120,6 +126,32 @@ function fixtureTree(): PlanTree {
);
}
/**
* Findings shaped exactly like what the extractor would produce for a
* session's plan tree: one proposition per tree node (statement = title,
* treeNodeId set), one `requires` edge per tree edge.
*/
function treeFindings(tree: PlanTree): SessionFindings {
const idByNodeId = new Map(tree.nodes.map((n, i) => [n.id, `p${i + 1}`]));
return {
propositions: tree.nodes.map((n, i) => ({
id: `p${i + 1}`,
statement: n.title,
aliases: [],
kind: "truth" as const,
treeNodeId: n.id,
evidence: ["fixture evidence"],
})),
edges: tree.edges.map((e) => ({
from: idByNodeId.get(e.from)!,
to: idByNodeId.get(e.to)!,
type: "requires" as const,
evidence: ["fixture evidence"],
})),
attribution: { quiz: {} },
};
}
/** The full fixture transcript: probes → plan → checks → mark_done → noise. */
function fixtureMessages(tree: PlanTree): UIMessage[] {
return [
@@ -154,7 +186,8 @@ function fixtureMessages(tree: PlanTree): UIMessage[] {
),
]),
assistant("a6", [
// nodeId present but NOT in the tree → unmapped → null + question text
// nodeId present but NOT in the tree → tagged-but-unmapped: the
// attribution can rescue it, or it lands null + question text.
quizPart(
"call-5",
{ question: "Q5", nodeId: "ghost", options: [{ label: "a" }, { label: "b" }], correctAnswer: "a", explanation: "e" },
@@ -163,7 +196,7 @@ function fixtureMessages(tree: PlanTree): UIMessage[] {
]),
assistant("a7", [planPart("p2", "plan_mark_done", { id: "alpha" }, tree)]),
assistant("a8", [
// dismissed quiz + an answered ask — neither emits an event
// dismissed quiz + an answered ask — neither becomes an event
quizPart(
"call-6",
{ question: "Q6", options: [{ label: "a" }, { label: "b" }], correctAnswer: "a", explanation: "e" },
@@ -180,13 +213,27 @@ function fixtureMessages(tree: PlanTree): UIMessage[] {
];
}
/** The chat every emission test folds (shared, so ordering is intentional). */
/**
* The extractor's findings for the fixture chat: the tree as propositions +
* the probe attribution. The audit stage is a no-op pass-through in these
* merger tests — its behavior is tested with the pipeline (extraction.test).
*/
function fixtureFindings(): SessionFindings {
const findings = treeFindings(fixtureTree());
// Untagged probe quizzes: call-1 attributed retroactively to Alpha (the
// vision's open question — resolved by the extractor); call-2 genuinely
// unattributable; call-5's ghost nodeId is rescued onto Gamma.
findings.attribution.quiz = { "call-1": "p1", "call-2": null, "call-5": "p3" };
return findings;
}
/** The chat every merger test folds (shared, so ordering is intentional). */
const chat1 = fixtureChat("Graph fixture", fixtureTree(), fixtureMessages(fixtureTree()));
/** Set by the provenance test; the snapshot test (which runs after it) reads it. */
/** Set by the cross-session test; the snapshot test (which runs after it) reads it. */
let chat2Id = "";
// ─── emission ────────────────────────────────────────────────────────────────
// ─── the merger ──────────────────────────────────────────────────────────────
/** Everything the graph holds for a set of chats (other processes' test
* emissions run concurrently against the same files — scope, never assume). */
@@ -205,9 +252,9 @@ function scopedGraph(chatIds: string[]) {
};
}
describe("graph emission (emitChat)", () => {
it("folds tree nodes/edges and quiz/mark_done events with the probe/check boundary", () => {
const stats = emitChat(chat1.id, graphDeps);
describe("the merger (mergeFindings)", () => {
it("merges propositions/edges by identity and rebuilds events with retroactive probe attribution", () => {
const stats = mergeFindings(chat1.id, fixtureFindings(), graphDeps);
assert.deepEqual(stats, { nodes: 3, edges: 1, events: 6 });
const graph = scopedGraph([chat1.id]);
@@ -219,7 +266,7 @@ describe("graph emission (emitChat)", () => {
assert.match(node.id, /^n_/);
}
// Tree edge alpha→beta becomes a `requires` edge on the mapped nodes.
// The tree edge alpha→beta becomes a `requires` edge on the mapped nodes.
assert.equal(graph.edges.length, 1);
const edge = graph.edges[0]!;
assert.equal(edge.from, idByStatement.get("Alpha intuition"));
@@ -227,77 +274,69 @@ describe("graph emission (emitChat)", () => {
assert.equal(edge.type, "requires");
assert.deepEqual(edge.provenance, [chat1.id]);
// Events — kinds, refs, node attribution, notes, question fallback.
// Events — kinds, refs, notes, and the attribution paths.
assert.equal(graph.events.length, 6);
const byRef = new Map(graph.events.map((e) => [e.ref, e]));
const probeMiss = byRef.get("call-1")!;
assert.equal(probeMiss.kind, "probe-miss"); // quiz BEFORE plan_init
assert.equal(probeMiss.nodeId, null);
assert.equal(probeMiss.question, "Q1 text");
assert.equal(probeMiss.chatId, chat1.id);
assert.equal(probeMiss.nodeId, idByStatement.get("Alpha intuition"), "retroactive probe attribution");
assert.equal(probeMiss.question, undefined, "attributed → no question text needed");
const probeDontKnow = byRef.get("call-2")!;
assert.equal(probeDontKnow.kind, "dont-know"); // same kind, both phases
assert.equal(probeDontKnow.nodeId, null);
assert.equal(probeDontKnow.nodeId, null, "unattributable probe → null");
assert.equal(probeDontKnow.question, "Q2 text");
const check = byRef.get("call-3")!;
assert.equal(check.kind, "check-correct"); // quiz AFTER plan_init
assert.equal(check.nodeId, idByStatement.get("Alpha intuition"));
assert.equal(check.nodeId, idByStatement.get("Alpha intuition"), "tagged quiz → deterministic");
assert.equal(check.note, "recalled the diagram");
assert.equal(check.question, undefined); // resolved nodeId → no question text
const checkMiss = byRef.get("call-4")!;
assert.equal(checkMiss.kind, "check-incorrect");
assert.equal(checkMiss.nodeId, idByStatement.get("Beta rule"));
const unmapped = byRef.get("call-5")!;
assert.equal(unmapped.kind, "dont-know");
assert.equal(unmapped.nodeId, null); // nodeId present but unmapped
assert.equal(unmapped.question, "Q5 text");
const ghost = byRef.get("call-5")!;
assert.equal(ghost.kind, "dont-know");
assert.equal(ghost.nodeId, idByStatement.get("Gamma application"), "tagged-but-unmapped rescued by attribution");
assert.equal(ghost.question, undefined);
const taught = byRef.get("taught:alpha")!;
assert.equal(taught.kind, "taught");
assert.equal(taught.nodeId, idByStatement.get("Alpha intuition"));
assert.equal(taught.nodeId, idByStatement.get("Alpha intuition"), "mark_done → tree-node mapping");
// The dismissed quiz and the ask question emitted nothing.
// The dismissed quiz and the ask question became nothing.
assert.equal([...byRef.keys()].filter((r) => r === "call-6" || r === "call-7").length, 0);
});
it("is idempotent — re-emission appends nothing", () => {
const stats = emitChat(chat1.id, graphDeps);
it("is idempotent — re-merging the same findings appends nothing", () => {
const stats = mergeFindings(chat1.id, fixtureFindings(), graphDeps);
assert.deepEqual(stats, { nodes: 0, edges: 0, events: 0 });
const graph = scopedGraph([chat1.id]);
assert.equal(graph.nodes.length, 3);
assert.equal(graph.edges.length, 1);
assert.equal(graph.events.length, 6);
// exactly one checkpoint line per fold, though — checkpoints are a log
// exactly one checkpoint line per merge, though — checkpoints are a log
assert.equal(graphStore.readCheckpoints().filter((c) => c.chatId === chat1.id).length, 2);
});
it("accumulates edge provenance when another chat maps onto the same nodes", () => {
// Simulates what a future consolidation merge produces: chat2's tree
// slugs map onto chat1's graph nodes, so its edge assertion dedupes
// by (from,to,type) and just adds chat2 to the provenance.
it("merges a second session onto the same nodes and accumulates edge provenance", () => {
// chat2 re-teaches the same two propositions (identical statements) —
// the conservative identity resolution recognizes them.
const tree2 = createTree(
[
{ id: "alpha", title: "Alpha (second session)" },
{ id: "beta", title: "Beta (second session)" },
{ id: "a", title: "Alpha intuition" },
{ id: "b", title: "Beta rule" },
],
[{ from: "alpha", to: "beta" }],
[{ from: "a", to: "b" }],
);
const chat2 = fixtureChat("Graph fixture 2", tree2, [
user("u1", "again"),
assistant("a1", [planPart("p1", "plan_init", { nodes: [], edges: [] }, tree2)]),
]);
const before = getGraph(graphStore);
const gAlpha = before.nodes.find((n) => n.statement === "Alpha intuition")!.id;
const gBeta = before.nodes.find((n) => n.statement === "Beta rule")!.id;
graphStore.appendMapping({ chatId: chat2.id, treeNodeId: "alpha", graphNodeId: gAlpha });
graphStore.appendMapping({ chatId: chat2.id, treeNodeId: "beta", graphNodeId: gBeta });
const stats = emitChat(chat2.id, graphDeps);
const stats = mergeFindings(chat2.id, treeFindings(tree2), graphDeps);
assert.deepEqual(stats, { nodes: 0, edges: 1, events: 0 });
chat2Id = chat2.id;
@@ -312,7 +351,9 @@ describe("graph emission (emitChat)", () => {
const idByStatement = new Map(snap.nodes.map((n) => [n.statement, n.id]));
assert.equal(snap.nodeStates[idByStatement.get("Alpha intuition")!], "taught");
assert.equal(snap.nodeStates[idByStatement.get("Beta rule")!], "missed");
assert.equal(snap.nodeStates[idByStatement.get("Gamma application")!], "candidate");
// Gamma: no taught/checks — but the rescued dont-know probe (call-5)
// makes its latest event a dont-know.
assert.equal(snap.nodeStates[idByStatement.get("Gamma application")!], "dont-know");
assert.equal(snap.sessions[chat1.id], "Graph fixture");
assert.equal(snap.sessions[chat2Id], "Graph fixture 2");
@@ -320,44 +361,25 @@ describe("graph emission (emitChat)", () => {
(snap.nodeSessions[idByStatement.get("Alpha intuition")!] ?? []).sort(),
[chat1.id, chat2Id].sort(),
);
// Gamma was only ever created (mapping) by chat1.
// Gamma was only ever touched by chat1.
assert.deepEqual(snap.nodeSessions[idByStatement.get("Gamma application")!], [chat1.id]);
});
it("replays plan tool outputs when record.tree is missing (removed nodes stay out)", () => {
const initTree = createTree(
[
{ id: "x", title: "X live" },
{ id: "y", title: "Y pruned" },
],
[{ from: "x", to: "y" }],
);
const prunedTree = removeNode(initTree, "y");
const chat = fixtureChat("Replay fixture", null, [
user("u1", "go"),
assistant("a1", [planPart("p1", "plan_init", { nodes: [], edges: [] }, initTree)]),
assistant("a2", [planPart("p2", "plan_remove_node", { id: "y" }, prunedTree)]),
]);
const stats = emitChat(chat.id, graphDeps);
assert.deepEqual(stats, { nodes: 1, edges: 0, events: 0 }); // y pruned, edge cascaded away
const graph = scopedGraph([chat.id]);
assert.ok(graph.nodes.some((n) => n.statement === "X live"));
assert.ok(!graph.nodes.some((n) => n.statement === "Y pruned"));
});
it("returns zeros for unknown chats", () => {
assert.deepEqual(emitChat("chat_does_not_exist", graphDeps), { nodes: 0, edges: 0, events: 0 });
assert.deepEqual(mergeFindings("chat_does_not_exist", treeFindings(fixtureTree()), graphDeps), {
nodes: 0,
edges: 0,
events: 0,
});
});
});
// chat2 from the provenance test, reachable by the snapshot test below it.
// ─── the trigger helper ──────────────────────────────────────────────────────
describe("catch-up (foldStaleChats)", () => {
it("folds chats newer than their checkpoint, exactly once, deriving known state", () => {
describe("the trigger helper (pendingQuizAnswers)", () => {
it("counts answered quizzes not yet in the ledger — dismissed/ask never count", () => {
const tree = createTree([{ id: "solo", title: "Solo node" }], []);
const chat = fixtureChat("Catch-up fixture", tree, [
const chat = fixtureChat("Trigger fixture", tree, [
user("u1", "go"),
assistant("a1", [planPart("p1", "plan_init", { nodes: [], edges: [] }, tree)]),
assistant("a2", [
@@ -366,23 +388,39 @@ describe("catch-up (foldStaleChats)", () => {
{ question: "QS", nodeId: "solo", options: [{ label: "a" }, { label: "b" }], correctAnswer: "a", explanation: "e" },
answer({ questionId: "call-s", questionText: "QS text", correct: true, answeredAt: "2026-08-22T03:00:00.000Z" }),
),
quizPart(
"call-s2",
{ question: "QS2", options: [{ label: "a" }, { label: "b" }], correctAnswer: "a", explanation: "e" },
answer({ questionId: "call-s2", questionText: "QS2 text", dismissed: true }),
),
{
type: "tool-ask_user_question",
toolCallId: "call-s3",
state: "output-available",
input: { question: "free?" },
output: answer({ questionId: "call-s3", kind: "ask", questionText: "free?", text: "y" }),
},
]),
]);
const folded = foldStaleChats(graphDeps);
assert.ok(folded >= 1, "the catch-up fixture was folded");
assert.equal(graphStore.readCheckpoints().filter((c) => c.chatId === chat.id).length, 1);
assert.equal(pendingQuizAnswers(chat.id, graphDeps), 1, "one answered quiz pending");
assert.equal(pendingQuizAnswers("chat_does_not_exist", graphDeps), 0);
const snap = graphSnapshot(graphDeps);
const solo = snap.nodes.find((n) => n.statement === "Solo node");
assert.ok(solo, "solo node was emitted by the catch-up fold");
assert.equal(snap.nodeStates[solo.id], "known"); // correct check, no taught event
assert.equal(snap.sessions[chat.id], "Catch-up fixture");
mergeFindings(chat.id, treeFindings(tree), graphDeps);
assert.equal(pendingQuizAnswers(chat.id, graphDeps), 0, "extracted → nothing pending");
// Already checkpointed → not folded again (other test files' chats may
// legitimately fold here, so assert on THIS chat only).
foldStaleChats(graphDeps);
assert.equal(graphStore.readCheckpoints().filter((c) => c.chatId === chat.id).length, 1);
// A NEW answered quiz raises the count again (the re-trigger path).
chatStore.saveMessages(chat.id, [
...chatStore.getChat(chat.id)!.messages,
assistant("a3", [
quizPart(
"call-s4",
{ question: "QS4", nodeId: "solo", options: [{ label: "a" }, { label: "b" }], correctAnswer: "a", explanation: "e" },
answer({ questionId: "call-s4", questionText: "QS4 text", correct: false, answeredAt: "2026-08-22T03:01:00.000Z" }),
),
]),
]);
assert.equal(pendingQuizAnswers(chat.id, graphDeps), 1);
});
});