Extraction pipeline: extractor → auditor → merger — the sole graph writer (stage 5)
The knowledge graph's write channel is now the pivot's target
architecture (vision §1 construction):
- src/agent/subagents/extraction.ts — the pipeline factory
(createExtractionPipeline({ chatStore, graphStore, extractor?,
auditor? }); models env LH_EXTRACT_MODEL ?? LH_MODEL):
* transcriptView(): the compact view both LLM stages see — final plan
tree with node ids, conversation verbatim, QUIZ lines with outcomes,
mark_done lines; job tool parts are one-liners, plan ops fold into
the tree line.
* Extractor: transcript → SessionFindings (propositions incl.
misconceptions and prose-only supports edges, attribution for
UNTAGGED probes, all with verbatim evidence) via one zod-validated
submit_findings call.
* Auditor: transcript + findings → the verified subset + drops (the
faithfulness net) via submit_review.
- src/core/graph.ts — emitChat/foldStaleChats/replayFinalTree retire;
mergeFindings() replaces them as the pipeline's deterministic apply
stage: exact/alias identity resolution (one mapping line per (chat,
graph node)), edge dedupe, and event rebuild FROM THE TRANSCRIPT's tool
parts — kinds/timestamps/refs/notes deterministic, the LLM's only event
say being WHICH proposition an untagged probe tested (retroactive —
the vision's open question #3, resolved structurally: tagged events
resolve deterministically; attribution rescues untagged and
tagged-but-unmapped ones; unattributable land null + question text).
pendingQuizAnswers() is the trigger helper.
- Trigger (turn-service.ts triggerExtraction, replacing the old
foldTurnIntoGraph body): once a chat accumulates
EXTRACTION.everyNQuizAnswers (LH_EXTRACT_N, default 3) new answered
quizzes, fire the pipeline async — single-flight per chat, failure
checkpoints nothing and re-runs on the next trigger. Deliberately a v1
placeholder isolated in ONE seam, trigger-agnostic pipeline.
- GET /api/graph is now a pure read (foldStaleChats gone).
- NodeMapping.treeNodeId → sourceId (the mapping key is now the writer's
local proposition id — tree node when tree-derived); store strict-read
schema updated. Local ledger data regenerates through the pipeline.
- Tests: graph.test.ts rewritten around mergeFindings (identity,
attribution paths incl. ghost-rescue, idempotency, cross-session
provenance, trigger counting), graph-context.test.ts merge section
converted to findings-driven, NEW extraction.test.ts — transcriptView
units + the full pipeline with mock models (audit drops, failure
checkpoints nothing, single-flight). 195/195, tsc/eslint/build clean.
Probe attribution (vision open question): probes attach retroactively by
the extractor against the whole transcript; a pre-tree probe maps to a
proposition the tree only articulates later. Fuzzy/embedding merge
adjudication stays deferred (conservative exact/alias only).
This commit is contained in:
@@ -29,7 +29,8 @@ npx tsx scripts/bench-graph.ts # force-layout benchmark (synthetic graphs)
|
|||||||
|---|---|
|
|---|---|
|
||||||
| `OPENROUTER_API_KEY` | required — all LLM calls go through OpenRouter |
|
| `OPENROUTER_API_KEY` | required — all LLM calls go through OpenRouter |
|
||||||
| `LH_MODEL` | teacher model (default `anthropic/claude-sonnet-4.5`) |
|
| `LH_MODEL` | teacher model (default `anthropic/claude-sonnet-4.5`) |
|
||||||
| `LH_RESEARCH_MODEL`, `LH_DEMO_MODEL` | per-sub-agent overrides (fall back to `LH_MODEL`) |
|
| `LH_RESEARCH_MODEL`, `LH_DEMO_MODEL`, `LH_EXTRACT_MODEL` | per-sub-agent overrides (fall back to `LH_MODEL`) |
|
||||||
|
| `LH_EXTRACT_N` | answered quizzes per chat before the extraction pipeline fires (default `3`) |
|
||||||
| `LH_VISUAL_MODEL` | visual-maker model — must be **vision-capable** (it looks at rendered SVGs) |
|
| `LH_VISUAL_MODEL` | visual-maker model — must be **vision-capable** (it looks at rendered SVGs) |
|
||||||
| `GOOGLE_SEARCH_API_KEY` + `GOOGLE_CSE_ID` | researcher's `web_search` (Google Custom Search) |
|
| `GOOGLE_SEARCH_API_KEY` + `GOOGLE_CSE_ID` | researcher's `web_search` (Google Custom Search) |
|
||||||
| `CHROME_PATH` | optional explicit Chrome path for headless verification (puppeteer-core uses system Chrome; nothing is downloaded) |
|
| `CHROME_PATH` | optional explicit Chrome path for headless verification (puppeteer-core uses system Chrome; nothing is downloaded) |
|
||||||
@@ -58,7 +59,7 @@ graph TB
|
|||||||
end
|
end
|
||||||
|
|
||||||
subgraph TEACHER["Teacher agent (src/agent)"]
|
subgraph TEACHER["Teacher agent (src/agent)"]
|
||||||
CH["chat-handler.ts — turn composition root<br/>(services: context · persist · spawn · fold)"]
|
CH["chat-handler.ts — turn composition root<br/>(services: context · persist · spawn · extract)"]
|
||||||
TA["ToolLoopAgent<br/>system-prompt.md + prompts/*.md fragments<br/>+ graph-context block"]
|
TA["ToolLoopAgent<br/>system-prompt.md + prompts/*.md fragments<br/>+ graph-context block"]
|
||||||
QT["quiz · ask_user_question<br/>(client-side tools, no execute)"]
|
QT["quiz · ask_user_question<br/>(client-side tools, no execute)"]
|
||||||
PT["plan_* tools (server-side,<br/>mutate the chat's tree)"]
|
PT["plan_* tools (server-side,<br/>mutate the chat's tree)"]
|
||||||
@@ -69,6 +70,7 @@ graph TB
|
|||||||
RES["researcher<br/>web_search + web_fetch"]
|
RES["researcher<br/>web_search + web_fetch"]
|
||||||
DEMO["demo-builder<br/>write → headless-run → fix"]
|
DEMO["demo-builder<br/>write → headless-run → fix"]
|
||||||
VIS["visual-maker<br/>write → render → look"]
|
VIS["visual-maker<br/>write → render → look"]
|
||||||
|
EXT["extraction pipeline<br/>extractor → auditor → merger<br/>(the sole graph writer)"]
|
||||||
end
|
end
|
||||||
|
|
||||||
subgraph STORE["File stores (src/store adapters → data/, never in git)"]
|
subgraph STORE["File stores (src/store adapters → data/, never in git)"]
|
||||||
@@ -77,10 +79,10 @@ graph TB
|
|||||||
LEDG[("data/graph/*.jsonl<br/>nodes/edges/events/mappings/sessions")]
|
LEDG[("data/graph/*.jsonl<br/>nodes/edges/events/mappings/sessions")]
|
||||||
end
|
end
|
||||||
|
|
||||||
subgraph PIPE["Graph pipeline (src/core)"]
|
subgraph PIPE["Graph core (src/core) — merger + derivation"]
|
||||||
EMIT["graph.ts emitChat<br/>naive-union, idempotent,<br/>exact/alias match"]
|
MERGE["graph.ts mergeFindings<br/>deterministic apply: identity,<br/>event rebuild, dedupe, checkpoint"]
|
||||||
GCTX["graph-context.ts<br/>hotspots + 1-hop when large"]
|
GCTX["graph-context.ts<br/>hotspots + 1-hop when large"]
|
||||||
SNAP["graphSnapshot · foldStaleChats"]
|
SNAP["graphSnapshot (pure read)"]
|
||||||
end
|
end
|
||||||
|
|
||||||
TR & COMP & DOCK --> RCHAT
|
TR & COMP & DOCK --> RCHAT
|
||||||
@@ -95,17 +97,18 @@ graph TB
|
|||||||
PT --> CHATS
|
PT --> CHATS
|
||||||
CH -->|"onFinish: saveMessages"| CHATS
|
CH -->|"onFinish: saveMessages"| CHATS
|
||||||
CH -->|"onFinish: spawnPendingJobs"| RES & DEMO & VIS
|
CH -->|"onFinish: spawnPendingJobs"| RES & DEMO & VIS
|
||||||
CH -->|"onFinish: emitChat"| EMIT
|
CH -->|"onFinish: every N quiz answers<br/>(turn-service seam)"| EXT
|
||||||
|
|
||||||
RES & DEMO & VIS --> JOBS
|
RES & DEMO & VIS --> JOBS
|
||||||
RES & DEMO & VIS -->|"rewrite tool part<br/>→ output-available"| CHATS
|
RES & DEMO & VIS -->|"rewrite tool part<br/>→ output-available"| CHATS
|
||||||
|
|
||||||
EMIT --> LEDG
|
EXT -->|"audit-passed findings"| MERGE
|
||||||
|
MERGE --> LEDG
|
||||||
LEDG --> GCTX -->|"appended to instructions"| TA
|
LEDG --> GCTX -->|"appended to instructions"| TA
|
||||||
LEDG --> SNAP --> RGRAPH
|
LEDG --> SNAP --> RGRAPH
|
||||||
```
|
```
|
||||||
|
|
||||||
**Core invariant, carried over from the original design:** the graph is never asserted by an LLM post-hoc guesswork — v1 emission is deterministic code over structured tool data. The teacher stays a teacher; everything below the ledgers is derived.
|
**Core invariant:** the extraction pipeline (extractor → auditor → merger) is the sole graph writer. The LLM stages propose and verify (propositions, edges, probe attribution — each with verbatim evidence); the deterministic merger owns every ledger append and rebuilds anything the model could silently mis-copy (event kinds, timestamps, idempotency refs) from the transcript. The teacher stays a teacher; everything below the ledgers is derived.
|
||||||
|
|
||||||
## Layout
|
## Layout
|
||||||
|
|
||||||
@@ -124,16 +127,18 @@ src/
|
|||||||
tools/web.ts web_search (Google CSE) + web_fetch (Readability)
|
tools/web.ts web_search (Google CSE) + web_fetch (Readability)
|
||||||
tools/jobs.ts research / demo / visual tool surfaces
|
tools/jobs.ts research / demo / visual tool surfaces
|
||||||
subagents/ researcher · demo-builder · visual-maker,
|
subagents/ researcher · demo-builder · visual-maker,
|
||||||
|
extraction pipeline (extractor/auditor + prompts),
|
||||||
runner (async protocol), headless Chrome helper
|
runner (async protocol), headless Chrome helper
|
||||||
server/chat-handler.ts one chat turn: compose agent + services, stream
|
server/chat-handler.ts one chat turn: compose agent + services, stream
|
||||||
server/turn-service.ts turn services: teacher context · persist ·
|
server/turn-service.ts turn services: teacher context · persist ·
|
||||||
spawn jobs · fold-into-graph (error-contained)
|
spawn jobs · extraction trigger (the v1
|
||||||
|
quiz-answer threshold — one swappable seam)
|
||||||
server/stores.ts THE composition root — the one place adapters wire in
|
server/stores.ts THE composition root — the one place adapters wire in
|
||||||
core/contracts.ts THE contracts file — types + protocol docs
|
core/contracts.ts THE contracts file — types + protocol docs
|
||||||
core/ports.ts storage ports — ChatStore · GraphStore · JobStore
|
core/ports.ts storage ports — ChatStore · GraphStore · JobStore
|
||||||
(core code depends on these types only)
|
(core code depends on these types only)
|
||||||
core/ question-core, plan-tree, phases, pending,
|
core/ question-core, plan-tree, phases, pending,
|
||||||
graph (emission/derivation), graph-context, math delims
|
graph (merger/derivation), graph-context, math delims
|
||||||
— all pure: storage arrives as parameters
|
— all pure: storage arrives as parameters
|
||||||
store/ fs adapters implementing the ports: chats ·
|
store/ fs adapters implementing the ports: chats ·
|
||||||
graph ledgers · jobs (plain fs, no framework)
|
graph ledgers · jobs (plain fs, no framework)
|
||||||
@@ -149,15 +154,15 @@ data/ chats/ · graph/ · jobs/ — runtime state, git
|
|||||||
|
|
||||||
**The async sub-agent protocol.** `research` / `demo` / `visual` are also client-side tools: the server turn ends at the call, `onFinish` spawns a background runner (fire-and-forget), the job store tracks a live activity line, the client polls `/api/jobs` while parts are pending, and on completion the runner rewrites the tool part in the chat record to `output-available` — the client re-syncs, the continuation fires, the teacher receives the result as a normal tool output. Sub-agents **verify, never one-shot**: the demo-builder headlessly executes its HTML in system Chrome before delivery; the visual-maker renders each SVG and looks at it (needs a vision-capable `LH_VISUAL_MODEL`).
|
**The async sub-agent protocol.** `research` / `demo` / `visual` are also client-side tools: the server turn ends at the call, `onFinish` spawns a background runner (fire-and-forget), the job store tracks a live activity line, the client polls `/api/jobs` while parts are pending, and on completion the runner rewrites the tool part in the chat record to `output-available` — the client re-syncs, the continuation fires, the teacher receives the result as a normal tool output. Sub-agents **verify, never one-shot**: the demo-builder headlessly executes its HTML in system Chrome before delivery; the visual-maker renders each SVG and looks at it (needs a vision-capable `LH_VISUAL_MODEL`).
|
||||||
|
|
||||||
**The live graph.** Every completed turn folds the chat into append-only JSONL ledgers (`emitChat`: tree nodes/edges, quiz outcome events, taught events — idempotent, checkpointed, so partial sessions still deposit value). The home view derives node states from the ledgers and lays them out with forceAtlas2 (run-then-freeze — no idle simulation). At session start the derived graph is rendered into a compact markdown block appended to the teacher's instructions (`buildGraphContext`) — whole graph while small, hotspots + 1-hop neighborhood past 150 nodes.
|
**The knowledge graph (the two channels).** The write channel: the extraction pipeline — extractor (LLM: propositions, edges, probe attribution, each with verbatim evidence) → auditor (LLM: drops anything unbacked) → merger (`mergeFindings`, deterministic: exact/alias identity resolution, event rebuild from the transcript's tool parts, dedupe, checkpoint). Triggered once a chat accumulates `EXTRACTION.everyNQuizAnswers` (default 3, `LH_EXTRACT_N`) new answered quizzes — a v1 placeholder trigger isolated in one seam (`turn-service.ts`), swappable without touching the pipeline. Failure checkpoints nothing; the next trigger re-runs. The read channel: the home view derives node states from the ledgers (`GET /api/graph` is a pure read — forceAtlas2 run-then-freeze, no idle simulation), and at session start the derived graph is rendered into a compact markdown block appended to the teacher's instructions (`buildGraphContext`) — whole graph while small, hotspots + 1-hop neighborhood past 150 nodes.
|
||||||
|
|
||||||
## Testing
|
## Testing
|
||||||
|
|
||||||
All deterministic, no LLM (models are injected as `MockLanguageModelV4` where an agent loop is under test):
|
All deterministic, no LLM (models are injected as `MockLanguageModelV4` where an agent loop is under test):
|
||||||
|
|
||||||
1. **Pure core units** — plan-tree ops, question grading, phases, graph emission, graph-context selection, math delimiters.
|
1. **Pure core units** — plan-tree ops, question grading, phases, graph merger + trigger helper, graph-context selection, math delimiters.
|
||||||
2. **Server flows** — `chat-handler` driven directly with a scripted mock model: persistence, job spawning, part rewriting, continuation.
|
2. **Server flows** — `chat-handler` driven directly with a scripted mock model: persistence, job spawning, part rewriting, continuation.
|
||||||
3. **Sub-agent loops** — researcher/demo-builder/visual-maker with mock models + fake headless API.
|
3. **Sub-agent loops** — researcher/demo-builder/visual-maker with mock models + fake headless API; the extraction pipeline end-to-end with mock models (extractor → auditor → in-memory ledgers).
|
||||||
4. **Render tests** — transcript, cards, dock, composer, tree panel via React server-side rendering.
|
4. **Render tests** — transcript, cards, dock, composer, tree panel via React server-side rendering.
|
||||||
|
|
||||||
## Lineage
|
## Lineage
|
||||||
|
|||||||
@@ -15,7 +15,17 @@ export const PATHS = {
|
|||||||
} as const;
|
} as const;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The LIVE graph store (src/store/graph.ts) — owned by the app's emission
|
* The extraction pipeline (vision §1 construction): the sole graph writer.
|
||||||
|
* Triggered per chat once this many NEW quiz answers have landed since the
|
||||||
|
* chat's last extraction checkpoint (LH_EXTRACT_N). The trigger mechanism is
|
||||||
|
* a v1 placeholder — deliberately isolated behind one seam
|
||||||
|
* (server/turn-service.ts) so it can be swapped without touching the
|
||||||
|
* pipeline.
|
||||||
|
*/
|
||||||
|
export const EXTRACTION = { everyNQuizAnswers: Number(process.env.LH_EXTRACT_N ?? 3) } as const;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The LIVE graph store (src/store/graph.ts) — owned by the app's extraction
|
||||||
* pipeline. Line shapes: src/core/contracts.ts.
|
* pipeline. Line shapes: src/core/contracts.ts.
|
||||||
*/
|
*/
|
||||||
export const LEDGER_FILES = {
|
export const LEDGER_FILES = {
|
||||||
|
|||||||
@@ -0,0 +1,309 @@
|
|||||||
|
import { ToolLoopAgent, stepCountIs, tool, type LanguageModel, type UIMessage } from "ai";
|
||||||
|
import { openrouter } from "@openrouter/ai-sdk-provider";
|
||||||
|
import { z } from "zod";
|
||||||
|
import type { AuditResult, ChatRecord, SessionFindings } from "../../core/contracts.ts";
|
||||||
|
import { mergeFindings, type EmissionStats, type GraphDeps } from "../../core/graph.ts";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The extraction pipeline (vision §1 construction) — the sole graph writer.
|
||||||
|
* Three stages, one orchestrator:
|
||||||
|
*
|
||||||
|
* extractor (LLM) transcript → SessionFindings (propositions, edges,
|
||||||
|
* event attribution — each with verbatim evidence)
|
||||||
|
* auditor (LLM) findings → AuditResult (the verified subset + drops)
|
||||||
|
* merger (core) mergeFindings() → deterministic ledger appends
|
||||||
|
*
|
||||||
|
* The LLM stages supply only the judgment calls; the deterministic rebuild
|
||||||
|
* in the merger owns everything that must never be mis-copied (event kinds,
|
||||||
|
* timestamps, refs). Both agents are single-tool-call workers: their whole
|
||||||
|
* deliverable is one zod-validated submit call — a ToolLoopAgent purely so
|
||||||
|
* the model can retry a schema-invalid call.
|
||||||
|
*
|
||||||
|
* Created via createExtractionPipeline(deps) — the composition root
|
||||||
|
* (chat-handler) injects the storage ports + models; this module never
|
||||||
|
* imports an adapter. runExtraction is single-flight per chat: a second
|
||||||
|
* trigger while one run is in flight is a no-op (the next trigger picks up
|
||||||
|
* whatever the in-flight run missed). On failure nothing is checkpointed —
|
||||||
|
* the next trigger re-runs the whole thing.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const DEFAULT_EXTRACTION_MODEL =
|
||||||
|
process.env.LH_EXTRACT_MODEL ?? process.env.LH_MODEL ?? "anthropic/claude-sonnet-4.5";
|
||||||
|
|
||||||
|
// ─── the transcript view (what the LLM sees) ────────────────────────────────
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A quiz tool part's outcome, one line: the attribution evidence.
|
||||||
|
* (question text + what the learner did — kind/correct are rebuilt by the
|
||||||
|
* merger; the extractor only needs enough to attribute.)
|
||||||
|
*/
|
||||||
|
function quizLine(part: { toolCallId: string; input: unknown; output: unknown }): string {
|
||||||
|
const input = part.input as { question?: unknown } | undefined;
|
||||||
|
const out = part.output as Partial<{ questionText: string; correct: boolean; dontKnow: boolean; note: string }> | undefined;
|
||||||
|
const question = typeof input?.question === "string" ? input.question : out?.questionText ?? "(no question)";
|
||||||
|
const outcome = out?.dontKnow === true ? "did not know" : out?.correct === true ? "answered correctly" : "answered incorrectly";
|
||||||
|
const note = typeof out?.note === "string" && out.note ? ` (note: ${out.note})` : "";
|
||||||
|
return `QUIZ [${part.toolCallId}] ${question} — the learner ${outcome}${note}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The compact transcript the extractor + auditor both see: user/teacher
|
||||||
|
* text verbatim (evidence quotes must match it), the final plan tree, and
|
||||||
|
* the structured tool events (quiz outcomes, mark_done). Job tool parts
|
||||||
|
* (research/demo/visual) are noise for extraction — their outcomes are
|
||||||
|
* summarized as one line each.
|
||||||
|
*/
|
||||||
|
export function transcriptView(record: ChatRecord): string {
|
||||||
|
const lines: string[] = [];
|
||||||
|
lines.push(`SESSION: ${record.title}`);
|
||||||
|
|
||||||
|
if (record.tree && record.tree.nodes.length > 0) {
|
||||||
|
const nodes = record.tree.nodes.map((n) => `${n.id} = ${n.title}`).join("; ");
|
||||||
|
lines.push(`FINAL PLAN TREE (node ids): ${nodes}`);
|
||||||
|
if (record.tree.edges.length > 0) {
|
||||||
|
const edges = record.tree.edges.map((e) => `${e.from} requires ${e.to}`).join("; ");
|
||||||
|
lines.push(`TREE EDGES: ${edges}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const message of record.messages as UIMessage[]) {
|
||||||
|
const who = message.role === "user" ? "LEARNER" : "TEACHER";
|
||||||
|
for (const part of message.parts) {
|
||||||
|
if (part.type === "text" && part.text.trim()) {
|
||||||
|
lines.push(`${who}: ${part.text.trim()}`);
|
||||||
|
} else if (part.type.startsWith("tool-")) {
|
||||||
|
const name = part.type.slice("tool-".length);
|
||||||
|
const p = part as unknown as { toolCallId: string; state: string; input: unknown; output: unknown };
|
||||||
|
if (name === "quiz" && p.state === "output-available") {
|
||||||
|
lines.push(quizLine(p));
|
||||||
|
} else if (name === "plan_mark_done" && p.state === "output-available") {
|
||||||
|
const input = p.input as { id?: unknown } | undefined;
|
||||||
|
lines.push(`TEACHER marked tree node done [${p.toolCallId}]: ${typeof input?.id === "string" ? input.id : "?"}`);
|
||||||
|
} else if (name.startsWith("plan_")) {
|
||||||
|
// plan ops are folded into the FINAL PLAN TREE line above
|
||||||
|
} else if (p.state === "output-available") {
|
||||||
|
lines.push(`(${name} tool result delivered)`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return lines.join("\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── the extractor ──────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
const evidence = z
|
||||||
|
.array(z.string())
|
||||||
|
.min(1)
|
||||||
|
.describe("Verbatim quotes from the transcript that back this. The auditor will check every quote.");
|
||||||
|
|
||||||
|
const findingsSchema = z.object({
|
||||||
|
propositions: z
|
||||||
|
.array(
|
||||||
|
z.object({
|
||||||
|
id: z.string().describe("Local id: \"p1\", \"p2\", … — referenced by edges and attribution."),
|
||||||
|
statement: z.string().describe("The proposition as one reasoning step — a node statement."),
|
||||||
|
aliases: z.array(z.string()).describe("Verbatim alternate phrasings from this session, if any."),
|
||||||
|
kind: z.enum(["truth", "misconception"]).describe("misconception = a wrong idea the learner holds or was warned about."),
|
||||||
|
treeNodeId: z.string().optional().describe("The plan-tree node id this came from, when it did."),
|
||||||
|
evidence,
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.describe("Every distinct proposition the session touched — from the plan tree AND from prose."),
|
||||||
|
edges: z
|
||||||
|
.array(
|
||||||
|
z.object({
|
||||||
|
from: z.string().describe("Proposition id that is the prerequisite / support."),
|
||||||
|
to: z.string().describe("Proposition id that depends on / is supported by `from`."),
|
||||||
|
type: z.enum(["requires", "supports"]),
|
||||||
|
evidence,
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.describe("Relations between propositions. `requires` = prerequisite; `supports` = helpful but not prerequisite. Include relations the plan tree does NOT show when the transcript articulates them."),
|
||||||
|
attribution: z
|
||||||
|
.object({
|
||||||
|
quiz: z
|
||||||
|
.record(z.string(), z.string().nullable())
|
||||||
|
.describe("toolCallId of every UNTAGGED quiz line → the proposition id it was really testing. Probe questions asked BEFORE the tree existed are attributed RETROACTIVELY — the whole transcript is visible; map them to the proposition they were probing (often articulated only later). null when genuinely unattributable. Tagged quizzes (with a tree nodeId in the transcript view) resolve automatically — do not include them."),
|
||||||
|
})
|
||||||
|
.describe("Probe attribution — your only event judgment."),
|
||||||
|
});
|
||||||
|
|
||||||
|
const EXTRACTOR_INSTRUCTIONS = `You are the extraction specialist in a learning application. You read ONE session
|
||||||
|
transcript and report what was learned — as structured findings for the learner's global knowledge graph.
|
||||||
|
|
||||||
|
The transcript view shows: the session's final plan tree (dependency map, with node ids), the conversation
|
||||||
|
verbatim (LEARNER / TEACHER), and structured events (QUIZ lines with the learner's outcome, mark_done lines).
|
||||||
|
|
||||||
|
Report, with verbatim evidence quotes for everything:
|
||||||
|
|
||||||
|
1. PROPOSITIONS — every distinct reasoning-step-sized proposition the session touched:
|
||||||
|
- Plan-tree nodes become propositions (pass their treeNodeId) — statement from the node title, enriched to a
|
||||||
|
complete proposition if the prose elaborates it.
|
||||||
|
- Propositions that appeared only in prose (teacher explanations, learner questions) get fresh ids.
|
||||||
|
- Include MISCONCEPTIONS — wrong ideas the learner showed or the teacher warned against — with kind
|
||||||
|
"misconception".
|
||||||
|
- One reasoning step per proposition: never a topic blob, never a sub-step.
|
||||||
|
- aliases: verbatim alternate phrasings this session used for the same thing.
|
||||||
|
|
||||||
|
2. EDGES between propositions:
|
||||||
|
- The tree's dependency edges are the backbone: from = prerequisite, to = dependent, type "requires".
|
||||||
|
- Also include relations articulated only in prose. Use "supports" when the transcript presents something as
|
||||||
|
helpful but NOT a prerequisite.
|
||||||
|
|
||||||
|
3. ATTRIBUTION — UNTAGGED quizzes only:
|
||||||
|
- QUIZ lines whose question carries a tree nodeId resolve automatically; do not attribute those.
|
||||||
|
- The others are PROBE-phase questions (asked before the tree was initialized) that test prerequisite
|
||||||
|
understanding. Attribute them RETROACTIVELY to the proposition they were really probing — even if that
|
||||||
|
proposition is only articulated later in the session. The QUIZ question text is your guide. If you truly
|
||||||
|
cannot tell, use null.
|
||||||
|
|
||||||
|
Evidence discipline: every quote must appear VERBATIM in the transcript view (a QUIZ line counts as the
|
||||||
|
transcript). The auditor rejects anything unbacked. When in doubt, report; the auditor decides.
|
||||||
|
|
||||||
|
Your entire deliverable is ONE submit_findings call.`;
|
||||||
|
|
||||||
|
function createExtractor(model: LanguageModel) {
|
||||||
|
return new ToolLoopAgent({
|
||||||
|
model,
|
||||||
|
instructions: EXTRACTOR_INSTRUCTIONS,
|
||||||
|
tools: {
|
||||||
|
submit_findings: tool({
|
||||||
|
description: "Submit the session findings.",
|
||||||
|
inputSchema: findingsSchema,
|
||||||
|
execute: async (findings) => findings,
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
stopWhen: stepCountIs(4),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── the auditor ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
const auditSchema = findingsSchema.extend({
|
||||||
|
dropped: z
|
||||||
|
.array(
|
||||||
|
z.object({
|
||||||
|
ref: z.string().describe("Proposition id, \"from→to\" for an edge, or attribution toolCallId."),
|
||||||
|
reason: z.string(),
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.describe("Everything you dropped, with the reason — the audit trail."),
|
||||||
|
});
|
||||||
|
|
||||||
|
const AUDITOR_INSTRUCTIONS = `You are the audit stage of a learning application's extraction pipeline. You receive
|
||||||
|
ONE session transcript and a set of proposed findings (propositions, edges, event attribution). Your job is the
|
||||||
|
faithfulness net: verify every candidate against the transcript, keep only what is backed, and return the
|
||||||
|
verified subset.
|
||||||
|
|
||||||
|
Verify:
|
||||||
|
- PROPOSITIONS: every evidence quote must appear (near-)verbatim in the transcript view, and the statement must
|
||||||
|
be a faithful reading of it. Drop invented or bloated statements. Conservative: when two proposed propositions
|
||||||
|
are really one, keep the better-stated one and drop the other.
|
||||||
|
- EDGES: the evidence must actually articulate the relation, in the direction claimed (from = prerequisite).
|
||||||
|
Drop direction-swapped or speculative edges.
|
||||||
|
- ATTRIBUTION: only UNTAGGED quizzes carry attribution — the QUIZ line's question must genuinely test the
|
||||||
|
attributed proposition. Drop wrong attributions to null, or re-point them to the proposition the question
|
||||||
|
actually tests.
|
||||||
|
|
||||||
|
Also drop propositions not referenced by anything else ONLY if they have no other backing — an unreferenced but
|
||||||
|
well-evidenced proposition is still knowledge.
|
||||||
|
|
||||||
|
Never invent new candidates. Your entire deliverable is ONE submit_review call containing the verified findings
|
||||||
|
(the same shapes you received) plus the dropped list.`;
|
||||||
|
|
||||||
|
function createAuditor(model: LanguageModel) {
|
||||||
|
return new ToolLoopAgent({
|
||||||
|
model,
|
||||||
|
instructions: AUDITOR_INSTRUCTIONS,
|
||||||
|
tools: {
|
||||||
|
submit_review: tool({
|
||||||
|
description: "Submit the verified findings subset + the dropped list.",
|
||||||
|
inputSchema: auditSchema,
|
||||||
|
execute: async (review) => review,
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
stopWhen: stepCountIs(4),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── the pipeline ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export interface ExtractionModels {
|
||||||
|
/** The extractor's model (tests inject a MockLanguageModelV4). */
|
||||||
|
extractor?: LanguageModel;
|
||||||
|
/** The auditor's model. */
|
||||||
|
auditor?: LanguageModel;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ExtractionPipelineDeps extends GraphDeps, ExtractionModels {}
|
||||||
|
|
||||||
|
export interface ExtractionPipeline {
|
||||||
|
/** Extract + audit + merge ONE chat. Single-flight per chat; resolves to
|
||||||
|
* the merger's append stats (or throws — no checkpoint on failure). */
|
||||||
|
runExtraction(chatId: string): Promise<EmissionStats>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createExtractionPipeline(deps: ExtractionPipelineDeps): ExtractionPipeline {
|
||||||
|
const { chatStore, graphStore } = deps;
|
||||||
|
const extractorAgent = createExtractor(deps.extractor ?? openrouter(DEFAULT_EXTRACTION_MODEL));
|
||||||
|
const auditorAgent = createAuditor(deps.auditor ?? openrouter(DEFAULT_EXTRACTION_MODEL));
|
||||||
|
const inFlight = new Set<string>();
|
||||||
|
|
||||||
|
async function extract(chatId: string): Promise<SessionFindings> {
|
||||||
|
const record = chatStore.getChat(chatId);
|
||||||
|
if (!record) throw new Error(`Chat not found: ${chatId}`);
|
||||||
|
const view = transcriptView(record);
|
||||||
|
|
||||||
|
const extraction = await extractorAgent.generate({ prompt: `Transcript:\n\n${view}` });
|
||||||
|
for (const part of extraction.content) {
|
||||||
|
if (part.type === "tool-result" && part.toolName === "submit_findings") {
|
||||||
|
return part.output as SessionFindings;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
throw new Error("the extractor never submitted findings");
|
||||||
|
}
|
||||||
|
|
||||||
|
async function audit(chatId: string, findings: SessionFindings): Promise<SessionFindings> {
|
||||||
|
const record = chatStore.getChat(chatId);
|
||||||
|
if (!record) throw new Error(`Chat not found: ${chatId}`);
|
||||||
|
const view = transcriptView(record);
|
||||||
|
|
||||||
|
const review = await auditorAgent.generate({
|
||||||
|
prompt: `Transcript:\n\n${view}\n\nProposed findings:\n\n${JSON.stringify(findings, null, 2)}`,
|
||||||
|
});
|
||||||
|
let auditResult: AuditResult | null = null;
|
||||||
|
for (const part of review.content) {
|
||||||
|
if (part.type === "tool-result" && part.toolName === "submit_review") {
|
||||||
|
auditResult = part.output as AuditResult;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!auditResult) throw new Error("the auditor never submitted a review");
|
||||||
|
if (auditResult.dropped.length > 0) {
|
||||||
|
console.info(
|
||||||
|
`[extraction ${chatId}] audit dropped ${auditResult.dropped.length}: ` +
|
||||||
|
auditResult.dropped.map((d) => `${d.ref} (${d.reason})`).join("; "),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const { dropped: _dropped, ...verified } = auditResult;
|
||||||
|
return verified;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function runExtraction(chatId: string): Promise<EmissionStats> {
|
||||||
|
if (inFlight.has(chatId)) return { nodes: 0, edges: 0, events: 0 };
|
||||||
|
inFlight.add(chatId);
|
||||||
|
try {
|
||||||
|
const findings = await extract(chatId);
|
||||||
|
const verified = await audit(chatId, findings);
|
||||||
|
// The merger re-reads the chat record itself — events are rebuilt
|
||||||
|
// from the transcript, never from stale copies.
|
||||||
|
const stats = mergeFindings(chatId, verified, { chatStore, graphStore });
|
||||||
|
console.info(`[extraction ${chatId}] merged: ${stats.nodes} nodes, ${stats.edges} edges, ${stats.events} events`);
|
||||||
|
return stats;
|
||||||
|
} finally {
|
||||||
|
inFlight.delete(chatId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return { runExtraction };
|
||||||
|
}
|
||||||
+82
-13
@@ -83,12 +83,14 @@ export interface GraphEdge {
|
|||||||
provenance: string[];
|
provenance: string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
/** mappings.jsonl line — the naive-union identity map: a chat's tree node →
|
/** mappings.jsonl line — the session→graph identity map: the extraction
|
||||||
* the graph node it folded into (the merge audit trail; v1 never merges two
|
* pipeline's proposition (identified by its source id) → the graph node it
|
||||||
* sessions' tree nodes — W8 exact-title match aside). */
|
* merged into. The merge audit trail: one line per (chat, graph node). */
|
||||||
export interface NodeMapping {
|
export interface NodeMapping {
|
||||||
chatId: string;
|
chatId: string;
|
||||||
treeNodeId: string;
|
/** The writer's local id for the proposition — a plan-tree node id when
|
||||||
|
* tree-derived, else the extractor's proposition id ("p1", …). */
|
||||||
|
sourceId: string;
|
||||||
graphNodeId: string;
|
graphNodeId: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -125,18 +127,85 @@ export interface LedgerEvent {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Store format (W4 implements src/store/graph.ts): JSONL ledgers at
|
* Store format (src/store/graph.ts): JSONL ledgers at config.LEDGER_FILES
|
||||||
* config.LEDGER_FILES (nodes/edges/events/mappings/sessions). The derived
|
* (nodes/edges/events/mappings/sessions). The derived graph is rebuilt by
|
||||||
* graph is rebuilt by reading the ledgers — store events, derive state.
|
* reading the ledgers — store events, derive state. sessions.jsonl is the
|
||||||
* sessions.jsonl tracks which chats have been folded into the graph
|
* extraction checkpoint: which chats the extraction pipeline has processed,
|
||||||
* (emission checkpoint — never fold the same turn twice).
|
* and through which updatedAt (never extract the same turns twice).
|
||||||
*
|
*
|
||||||
* v1 emission (naive union, deterministic, no LLM): a chat's plan tree folds
|
* The EXTRACTION PIPELINE is the sole graph writer (extractor → auditor →
|
||||||
* in — node title → statement (aliases empty), tree edges → requires edges,
|
* merger): the extractor proposes propositions/edges/attribution with
|
||||||
* mark_done → "taught" events; quiz parts with input.nodeId → probe / check
|
* verbatim evidence, the auditor drops anything unbacked, and the merger
|
||||||
* events (+ note); quiz parts without nodeId → events with nodeId null.
|
* (core/graph.ts, deterministic) resolves identity against the existing
|
||||||
|
* ledgers and appends. See SessionFindings below.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
// ─── Extraction findings (extractor output → audit → merger input) ───────────
|
||||||
|
|
||||||
|
/** One proposed proposition — a node candidate. */
|
||||||
|
export interface ExtractedProposition {
|
||||||
|
/** Extractor-local id ("p1", …) referenced by edges + attribution. */
|
||||||
|
id: string;
|
||||||
|
/** Canonical one-reasoning-step statement. */
|
||||||
|
statement: string;
|
||||||
|
/** Verbatim alternate phrasings from the session (merge aliases). */
|
||||||
|
aliases: string[];
|
||||||
|
kind: "truth" | "misconception";
|
||||||
|
/** The plan-tree node this came from, when it did (mapping provenance). */
|
||||||
|
treeNodeId?: string;
|
||||||
|
/** Verbatim transcript quotes backing the proposition. */
|
||||||
|
evidence: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** One proposed relation between two propositions. */
|
||||||
|
export interface ExtractedEdge {
|
||||||
|
/** Proposition id that is the prerequisite / support. */
|
||||||
|
from: string;
|
||||||
|
/** Proposition id that depends on / is supported by `from`. */
|
||||||
|
to: string;
|
||||||
|
type: EdgeType;
|
||||||
|
/** Verbatim transcript quotes backing the relation. */
|
||||||
|
evidence: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The extractor's event attribution — its ONLY say about events. The kinds,
|
||||||
|
* timestamps, idempotency refs, notes and question texts are all rebuilt
|
||||||
|
* deterministically by the merger from the transcript's quiz / mark_done
|
||||||
|
* tool parts; the LLM supplies exactly the judgment call: WHICH proposition
|
||||||
|
* each one is about (vision open question: probes attribute retroactively,
|
||||||
|
* against the whole transcript — a pre-tree probe can map to a proposition
|
||||||
|
* the tree only articulates later). */
|
||||||
|
/** The extractor's event attribution — its ONLY say about events. Event
|
||||||
|
* kinds, timestamps, idempotency refs, notes and question texts are all
|
||||||
|
* rebuilt deterministically by the merger from the transcript's tool
|
||||||
|
* parts, and TAGGED events (quiz nodeId, mark_done tree node) resolve
|
||||||
|
* deterministically through the proposition mapping. The LLM supplies
|
||||||
|
* exactly the one judgment call that needs it — the vision's open
|
||||||
|
* probe-attribution question: WHICH proposition did an UNTAGGED probe
|
||||||
|
* test? Answered retroactively against the whole transcript — a pre-tree
|
||||||
|
* probe can map to a proposition the tree only articulates later. */
|
||||||
|
export interface ExtractionAttribution {
|
||||||
|
/** UNTAGGED quiz toolCallId → proposition id; missing / null =
|
||||||
|
* unattributed (the event lands with nodeId null + the question text,
|
||||||
|
* exactly like unattributable probes always did). */
|
||||||
|
quiz: Record<string, string | null>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The extractor's — and, filtered, the auditor's — output shape. */
|
||||||
|
export interface SessionFindings {
|
||||||
|
propositions: ExtractedProposition[];
|
||||||
|
edges: ExtractedEdge[];
|
||||||
|
attribution: ExtractionAttribution;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The auditor's verdict: the verified subset, plus the drops (for the log —
|
||||||
|
* the audit trail of what was rejected and why). */
|
||||||
|
export interface AuditResult extends SessionFindings {
|
||||||
|
/** Dropped candidates: proposition id, edge "from→to", or attribution
|
||||||
|
* toolCallId — with the reason. */
|
||||||
|
dropped: { ref: string; reason: string }[];
|
||||||
|
}
|
||||||
|
|
||||||
/** Per-node state class for the home view (vision §1 "Home page") — derived,
|
/** Per-node state class for the home view (vision §1 "Home page") — derived,
|
||||||
* never stored: taught if ANY taught event; else the LATEST quiz event decides
|
* never stored: taught if ANY taught event; else the LATEST quiz event decides
|
||||||
* (known / missed / dont-know); a node with no events is a candidate. */
|
* (known / missed / dont-know); a node with no events is a candidate. */
|
||||||
|
|||||||
+138
-135
@@ -1,24 +1,29 @@
|
|||||||
/**
|
/**
|
||||||
* Knowledge-graph emission + derivation (W4; vision §1 "One substrate",
|
* The knowledge graph: merger (the deterministic final stage of the
|
||||||
* contracts.ts "v1 emission").
|
* extraction pipeline — vision §1 construction) + derivation (reads).
|
||||||
*
|
*
|
||||||
* emitChat(chatId) folds ONE chat into the global graph — naive union,
|
* The extraction pipeline is the SOLE graph writer:
|
||||||
* deterministic, no LLM:
|
|
||||||
* plan-tree nodes → GraphNodes (statement = title, aliases empty;
|
|
||||||
* W8: a title exactly matching an existing statement
|
|
||||||
* or alias — case-insensitive, trimmed — maps onto the
|
|
||||||
* existing node instead of appending a duplicate)
|
|
||||||
* plan-tree edges → `requires` GraphEdges (from = prerequisite)
|
|
||||||
* answered quizzes → ledger events (probe-* before the first plan_init
|
|
||||||
* output, check / dont-know after; ref = toolCallId)
|
|
||||||
* plan_mark_done → "taught" events (ref = taught:<treeNodeId>)
|
|
||||||
*
|
*
|
||||||
* Idempotent + incremental: mappings dedupe nodes, (from,to,type) dedupes
|
* extractor (LLM) → SessionFindings: propositions, edges, and event
|
||||||
* edge assertions, chat-scoped refs dedupe events, and sessions.jsonl
|
* attribution, each with verbatim evidence
|
||||||
* checkpoints which chats have been folded. Re-emitting a chat appends
|
* auditor (LLM) → the verified subset (unbacked candidates dropped)
|
||||||
* nothing. Every function takes the storage ports (core/ports.ts) as
|
* merger (HERE) → mergeFindings(): deterministic ledger appends
|
||||||
* explicit parameters — pure core, no adapter imports; the composition
|
*
|
||||||
* root (src/server/stores.ts) wires the fs adapters in.
|
* The merger is deliberately the ONLY stage that touches the ledgers, and it
|
||||||
|
* is pure code: the LLM supplies exactly the judgment calls (what was
|
||||||
|
* taught, what depends on what, which proposition a quiz tested), while
|
||||||
|
* event kinds/timestamps/refs — anything the model could silently
|
||||||
|
* mis-copy — are rebuilt deterministically from the transcript's tool
|
||||||
|
* parts. Idempotent + incremental: statement/alias matching dedupes nodes,
|
||||||
|
* (from,to,type) dedupes edge assertions, chat-scoped refs dedupe events,
|
||||||
|
* one mapping line per (chat, graph node), and sessions.jsonl checkpoints
|
||||||
|
* which chats have been extracted (never the same turns twice).
|
||||||
|
*
|
||||||
|
* Derivation below (getGraph / deriveNodeStates / graphSnapshot) is reads
|
||||||
|
* only — everything is recomputed from the ledgers. Every function takes
|
||||||
|
* the storage ports (core/ports.ts) as explicit parameters — pure core, no
|
||||||
|
* adapter imports; the composition root (src/server/stores.ts) wires the fs
|
||||||
|
* adapters in.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { randomBytes } from "node:crypto";
|
import { randomBytes } from "node:crypto";
|
||||||
@@ -30,8 +35,8 @@ import type {
|
|||||||
LedgerEvent,
|
LedgerEvent,
|
||||||
LedgerEventKind,
|
LedgerEventKind,
|
||||||
NodeStateClass,
|
NodeStateClass,
|
||||||
PlanTree,
|
|
||||||
PlanToolResult,
|
PlanToolResult,
|
||||||
|
SessionFindings,
|
||||||
} from "./contracts.ts";
|
} from "./contracts.ts";
|
||||||
import type { QuestionAnswerData } from "./question-core.ts";
|
import type { QuestionAnswerData } from "./question-core.ts";
|
||||||
import type { ChatStore, GraphStore } from "./ports.ts";
|
import type { ChatStore, GraphStore } from "./ports.ts";
|
||||||
@@ -47,7 +52,7 @@ function newGraphNodeId(): string {
|
|||||||
return `n_${randomBytes(6).toString("hex")}`;
|
return `n_${randomBytes(6).toString("hex")}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── transcript scanning ─────────────────────────────────────────────────────
|
// ─── transcript scanning (shared by the merger + trigger) ─────────────────────
|
||||||
|
|
||||||
/** A flattened, order-preserving view of every tool part in a transcript. */
|
/** A flattened, order-preserving view of every tool part in a transcript. */
|
||||||
interface ToolPartView {
|
interface ToolPartView {
|
||||||
@@ -84,23 +89,7 @@ function planResult(part: ToolPartView): PlanToolResult | null {
|
|||||||
return out as PlanToolResult;
|
return out as PlanToolResult;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
// ─── the merger (extraction pipeline stage 3 — deterministic) ────────────────
|
||||||
* The FINAL live tree, replayed from plan tool outputs in message order.
|
|
||||||
* Fallback for records whose stored `tree` is missing (it is the same state
|
|
||||||
* saveTree persists, so the two normally agree); also proves the probe/check
|
|
||||||
* boundary scan below sees the same transcript the teacher wrote.
|
|
||||||
*/
|
|
||||||
function replayFinalTree(messages: UIMessage[]): PlanTree {
|
|
||||||
let tree: PlanTree = { nodes: [], edges: [], currentNodeId: null, doneNodeIds: [], version: 0 };
|
|
||||||
for (const part of toolParts(messages)) {
|
|
||||||
if (!part.name.startsWith("plan_")) continue;
|
|
||||||
const result = planResult(part);
|
|
||||||
if (result && result.ok && result.tree) tree = result.tree;
|
|
||||||
}
|
|
||||||
return tree;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ─── emission ────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
export interface EmissionStats {
|
export interface EmissionStats {
|
||||||
/** New graph nodes appended (and mappings with them). */
|
/** New graph nodes appended (and mappings with them). */
|
||||||
@@ -112,79 +101,86 @@ export interface EmissionStats {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Fold one chat into the graph ledgers. Idempotent — safe to call after every
|
* Merge one chat's audited findings into the graph ledgers — the pipeline's
|
||||||
* turn (chat-handler onFinish) and again on catch-up (GET /api/graph). Sync
|
* apply stage. Deterministic + idempotent: re-merging the same findings (or
|
||||||
* and fast: a handful of small-file reads plus one append per new record.
|
* a later re-extraction of the same chat) appends nothing but the new.
|
||||||
*
|
*
|
||||||
* The storage ports are explicit (tests pass in-memory ledgers; production
|
* Identity resolution (v1: conservative): a proposition whose statement or
|
||||||
* passes the fs adapters from src/server/stores.ts).
|
* alias EXACTLY matches (case-insensitive, trimmed) an existing node's
|
||||||
|
* statement or alias merges onto it; everything else is a new node. One
|
||||||
|
* mapping line per (chat, graph node) — the merge audit trail. Fuzzy /
|
||||||
|
* embedding adjudication stays deferred (vision §1 consolidation).
|
||||||
|
*
|
||||||
|
* Events are rebuilt deterministically from the transcript's quiz and
|
||||||
|
* plan_mark_done parts (kinds, timestamps, notes, refs). TAGGED events
|
||||||
|
* (quiz nodeId, mark_done tree node) resolve through the proposition
|
||||||
|
* mapping — deterministic; the findings' attribution is consulted ONLY for
|
||||||
|
* untagged probe quizzes: the retroactive probe-attribution resolution (a
|
||||||
|
* pre-tree probe maps to a proposition the tree only articulates later).
|
||||||
|
* Unattributed quizzes land with nodeId null + the question text, exactly
|
||||||
|
* like the unattributable ones always have.
|
||||||
*/
|
*/
|
||||||
export function emitChat(chatId: string, deps: GraphDeps): EmissionStats {
|
export function mergeFindings(chatId: string, findings: SessionFindings, deps: GraphDeps): EmissionStats {
|
||||||
const { graphStore: ledgers } = deps;
|
const { graphStore: ledgers, chatStore } = deps;
|
||||||
const record = deps.chatStore.getChat(chatId);
|
const record = chatStore.getChat(chatId);
|
||||||
if (!record) return { nodes: 0, edges: 0, events: 0 };
|
if (!record) return { nodes: 0, edges: 0, events: 0 };
|
||||||
const stats: EmissionStats = { nodes: 0, edges: 0, events: 0 };
|
const stats: EmissionStats = { nodes: 0, edges: 0, events: 0 };
|
||||||
|
|
||||||
// 1. This chat's existing mappings — the identity map is per-chat (v1 is
|
// 1. Identity resolution. mapping: proposition id (and tree node id)
|
||||||
// naive union: two sessions' tree nodes never merge into one graph node).
|
// → graph node id — from this chat's existing mapping lines, then
|
||||||
|
// extended as findings resolve. chatNodeIds: graph nodes this chat has
|
||||||
|
// already mapped onto — the (chat, graph node) mapping dedupe.
|
||||||
const mapping = new Map<string, string>();
|
const mapping = new Map<string, string>();
|
||||||
|
const chatNodeIds = new Set<string>();
|
||||||
for (const m of ledgers.readMappings()) {
|
for (const m of ledgers.readMappings()) {
|
||||||
if (m.chatId === chatId) mapping.set(m.treeNodeId, m.graphNodeId);
|
if (m.chatId !== chatId || typeof m.sourceId !== "string") continue;
|
||||||
|
mapping.set(m.sourceId, m.graphNodeId);
|
||||||
|
chatNodeIds.add(m.graphNodeId);
|
||||||
}
|
}
|
||||||
|
|
||||||
// 2. The tree state to fold. record.tree is the truth (it is the exact
|
// 2. Propositions → graph nodes. Exact statement/alias match against the
|
||||||
// state every plan tool persisted); the transcript replay covers records
|
// whole ledger first (cross-session reuse), then against propositions
|
||||||
// that predate tree persistence. Nodes REMOVED from the tree are simply
|
// created earlier in THIS merge (intra-findings dedupe).
|
||||||
// absent here — if they were emitted by an earlier fold, their mappings
|
|
||||||
// and ledger lines stay (the ledgers are append-only evidence).
|
|
||||||
const tree =
|
|
||||||
record.tree && record.tree.nodes.length > 0 ? record.tree : replayFinalTree(record.messages);
|
|
||||||
|
|
||||||
// 3. Tree nodes without a mapping → GraphNode + mapping. W8 merge
|
|
||||||
// improvement: a title that EXACTLY matches (case-insensitive, trimmed) an
|
|
||||||
// existing node's statement or alias maps onto that node instead of
|
|
||||||
// appending a duplicate — the mapping line is still appended so provenance
|
|
||||||
// stays honest. Fuzzy/embedding matching stays deferred (vision §1
|
|
||||||
// consolidation pipeline).
|
|
||||||
const norm = (s: string): string => s.trim().toLowerCase();
|
const norm = (s: string): string => s.trim().toLowerCase();
|
||||||
const existingByTitle = new Map<string, string>();
|
const existingByTitle = new Map<string, string>();
|
||||||
for (const node of ledgers.readNodes()) {
|
for (const node of ledgers.readNodes()) {
|
||||||
const statementKey = norm(node.statement);
|
for (const title of [node.statement, ...node.aliases]) {
|
||||||
if (statementKey && !existingByTitle.has(statementKey)) existingByTitle.set(statementKey, node.id);
|
const key = norm(title);
|
||||||
for (const alias of node.aliases) {
|
if (key && !existingByTitle.has(key)) existingByTitle.set(key, node.id);
|
||||||
const aliasKey = norm(alias);
|
|
||||||
if (aliasKey && !existingByTitle.has(aliasKey)) existingByTitle.set(aliasKey, node.id);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
const now = new Date().toISOString();
|
const now = new Date().toISOString();
|
||||||
for (const node of tree.nodes) {
|
for (const p of findings.propositions) {
|
||||||
if (mapping.has(node.id)) continue;
|
if (mapping.has(p.id)) continue;
|
||||||
const titleKey = norm(node.title);
|
const keys = [p.statement, ...p.aliases].map(norm).filter((k) => k.length > 0);
|
||||||
const existingId = titleKey ? existingByTitle.get(titleKey) : undefined;
|
const matchId = keys.map((k) => existingByTitle.get(k)).find((id) => id !== undefined);
|
||||||
if (existingId !== undefined) {
|
let graphNodeId: string;
|
||||||
// Exact match: reuse the node, record the mapping, append nothing.
|
if (matchId !== undefined) {
|
||||||
ledgers.appendMapping({ chatId, treeNodeId: node.id, graphNodeId: existingId });
|
graphNodeId = matchId; // reuse — conservative merge, duplicates over wrong merges
|
||||||
mapping.set(node.id, existingId);
|
} else {
|
||||||
continue;
|
const graphNode: GraphNode = {
|
||||||
|
id: newGraphNodeId(),
|
||||||
|
statement: p.statement,
|
||||||
|
aliases: [...p.aliases],
|
||||||
|
kind: p.kind,
|
||||||
|
createdAt: now,
|
||||||
|
};
|
||||||
|
ledgers.appendNode(graphNode);
|
||||||
|
stats.nodes++;
|
||||||
|
graphNodeId = graphNode.id;
|
||||||
}
|
}
|
||||||
const graphNode: GraphNode = {
|
for (const key of keys) if (!existingByTitle.has(key)) existingByTitle.set(key, graphNodeId);
|
||||||
id: newGraphNodeId(),
|
if (!chatNodeIds.has(graphNodeId)) {
|
||||||
// statement is the title; the summary stays out (contracts).
|
ledgers.appendMapping({ chatId, sourceId: p.treeNodeId ?? p.id, graphNodeId });
|
||||||
statement: node.title,
|
chatNodeIds.add(graphNodeId);
|
||||||
aliases: [],
|
}
|
||||||
kind: "truth",
|
mapping.set(p.id, graphNodeId);
|
||||||
createdAt: now,
|
if (p.treeNodeId !== undefined) mapping.set(p.treeNodeId, graphNodeId);
|
||||||
};
|
|
||||||
ledgers.appendNode(graphNode);
|
|
||||||
ledgers.appendMapping({ chatId, treeNodeId: node.id, graphNodeId: graphNode.id });
|
|
||||||
existingByTitle.set(titleKey, graphNode.id);
|
|
||||||
mapping.set(node.id, graphNode.id);
|
|
||||||
stats.nodes++;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// 4. Tree edges → `requires` edges. Dedupe by (from,to,type) across the
|
// 3. Edges. Dedupe by (from,to,type) across the whole ledger: an existing
|
||||||
// whole ledger: an existing assertion just gains this chatId as
|
// assertion just gains this chatId as provenance (appended as a new
|
||||||
// provenance (appended as a new line; reads merge).
|
// line; reads merge).
|
||||||
const asserted = new Map<string, Set<string>>();
|
const asserted = new Map<string, Set<string>>();
|
||||||
for (const line of ledgers.readEdges()) {
|
for (const line of ledgers.readEdges()) {
|
||||||
const key = `${line.from}\u0000${line.to}\u0000${line.type}`;
|
const key = `${line.from}\u0000${line.to}\u0000${line.type}`;
|
||||||
@@ -195,23 +191,25 @@ export function emitChat(chatId: string, deps: GraphDeps): EmissionStats {
|
|||||||
}
|
}
|
||||||
for (const c of line.provenance) provenance.add(c);
|
for (const c of line.provenance) provenance.add(c);
|
||||||
}
|
}
|
||||||
for (const edge of tree.edges) {
|
for (const edge of findings.edges) {
|
||||||
const from = mapping.get(edge.from);
|
const from = mapping.get(edge.from);
|
||||||
const to = mapping.get(edge.to);
|
const to = mapping.get(edge.to);
|
||||||
if (!from || !to || from === to) continue; // unmapped endpoint — skip
|
if (!from || !to || from === to) continue; // unmapped endpoint — skip
|
||||||
const key = `${from}\u0000${to}\u0000requires`;
|
const key = `${from}\u0000${to}\u0000${edge.type}`;
|
||||||
const provenance = asserted.get(key);
|
const provenance = asserted.get(key);
|
||||||
if (provenance?.has(chatId)) continue; // already asserted by this chat
|
if (provenance?.has(chatId)) continue; // already asserted by this chat
|
||||||
ledgers.appendEdge({ from, to, type: "requires", provenance: [chatId] });
|
ledgers.appendEdge({ from, to, type: edge.type, provenance: [chatId] });
|
||||||
if (provenance) provenance.add(chatId);
|
if (provenance) provenance.add(chatId);
|
||||||
else asserted.set(key, new Set([chatId]));
|
else asserted.set(key, new Set([chatId]));
|
||||||
stats.edges++;
|
stats.edges++;
|
||||||
}
|
}
|
||||||
|
|
||||||
// 5. Events, replayed in transcript order so the probe/check boundary is
|
// 4. Events, rebuilt deterministically in transcript order so the
|
||||||
// exact: quizzes BEFORE the first plan_init output are probes. Refs are
|
// probe/check boundary is exact: quizzes BEFORE the first plan_init
|
||||||
// chat-scoped (a quiz toolCallId or taught:<treeNodeId>) — a ref already
|
// output are probes. Refs are chat-scoped (a quiz toolCallId or
|
||||||
// in the ledger means this turn was folded before.
|
// taught:<treeNodeId>) — a ref already in the ledger means this turn
|
||||||
|
// was extracted before. The findings' attribution supplies the
|
||||||
|
// nodeId, nothing else.
|
||||||
const seenRefs = new Set<string>();
|
const seenRefs = new Set<string>();
|
||||||
for (const event of ledgers.readEvents()) {
|
for (const event of ledgers.readEvents()) {
|
||||||
if (event.chatId === chatId && event.ref) seenRefs.add(event.ref);
|
if (event.chatId === chatId && event.ref) seenRefs.add(event.ref);
|
||||||
@@ -223,7 +221,7 @@ export function emitChat(chatId: string, deps: GraphDeps): EmissionStats {
|
|||||||
if (part.name === "plan_init" && part.state === "output-available") planSeen = true;
|
if (part.name === "plan_init" && part.state === "output-available") planSeen = true;
|
||||||
|
|
||||||
if (part.name === "quiz" && part.state === "output-available") {
|
if (part.name === "quiz" && part.state === "output-available") {
|
||||||
const event = quizEvent(part, chatId, mapping, planSeen, now);
|
const event = quizEvent(part, chatId, mapping, findings, planSeen, now);
|
||||||
if (event?.ref !== undefined && !seenRefs.has(event.ref)) {
|
if (event?.ref !== undefined && !seenRefs.has(event.ref)) {
|
||||||
ledgers.appendEvent(event);
|
ledgers.appendEvent(event);
|
||||||
seenRefs.add(event.ref);
|
seenRefs.add(event.ref);
|
||||||
@@ -237,6 +235,7 @@ export function emitChat(chatId: string, deps: GraphDeps): EmissionStats {
|
|||||||
if (result?.ok && typeof input?.id === "string") {
|
if (result?.ok && typeof input?.id === "string") {
|
||||||
const ref = `taught:${input.id}`;
|
const ref = `taught:${input.id}`;
|
||||||
if (!seenRefs.has(ref)) {
|
if (!seenRefs.has(ref)) {
|
||||||
|
// mark_done is tagged with the tree node id — deterministic.
|
||||||
ledgers.appendEvent({
|
ledgers.appendEvent({
|
||||||
nodeId: mapping.get(input.id) ?? null,
|
nodeId: mapping.get(input.id) ?? null,
|
||||||
kind: "taught",
|
kind: "taught",
|
||||||
@@ -251,16 +250,18 @@ export function emitChat(chatId: string, deps: GraphDeps): EmissionStats {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// 6. Checkpoint: this chat is folded through its current updatedAt.
|
// 5. Checkpoint: this chat is extracted through its current updatedAt.
|
||||||
ledgers.appendCheckpoint({ chatId, foldedAt: record.updatedAt });
|
ledgers.appendCheckpoint({ chatId, foldedAt: record.updatedAt });
|
||||||
return stats;
|
return stats;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Build the ledger event for one answered quiz tool part (or null to skip). */
|
/** Build the ledger event for one answered quiz tool part (or null to skip).
|
||||||
|
* Deterministic in everything except the attribution-supplied nodeId. */
|
||||||
function quizEvent(
|
function quizEvent(
|
||||||
part: ToolPartView,
|
part: ToolPartView,
|
||||||
chatId: string,
|
chatId: string,
|
||||||
mapping: Map<string, string>,
|
mapping: Map<string, string>,
|
||||||
|
findings: SessionFindings,
|
||||||
planSeen: boolean,
|
planSeen: boolean,
|
||||||
now: string,
|
now: string,
|
||||||
): LedgerEvent | null {
|
): LedgerEvent | null {
|
||||||
@@ -275,9 +276,13 @@ function quizEvent(
|
|||||||
else if (answer.correct === true) kind = planSeen ? "check-correct" : "probe-correct";
|
else if (answer.correct === true) kind = planSeen ? "check-correct" : "probe-correct";
|
||||||
else kind = planSeen ? "check-incorrect" : "probe-miss";
|
else kind = planSeen ? "check-incorrect" : "probe-miss";
|
||||||
|
|
||||||
|
// Tagged (input.nodeId) resolves deterministically through the mapping;
|
||||||
|
// attribution is the LLM judgment call for UNTAGGED probes only — the
|
||||||
|
// vision's open probe-attribution question, resolved retroactively.
|
||||||
const input = part.input as { nodeId?: unknown } | undefined;
|
const input = part.input as { nodeId?: unknown } | undefined;
|
||||||
const treeNodeId = typeof input?.nodeId === "string" ? input.nodeId : null;
|
const tagged = typeof input?.nodeId === "string" ? mapping.get(input.nodeId) : undefined;
|
||||||
const nodeId = treeNodeId !== null ? (mapping.get(treeNodeId) ?? null) : null;
|
const attributedId = findings.attribution.quiz[part.toolCallId];
|
||||||
|
const nodeId = tagged ?? (attributedId ? (mapping.get(attributedId) ?? null) : null);
|
||||||
|
|
||||||
const event: LedgerEvent = {
|
const event: LedgerEvent = {
|
||||||
nodeId,
|
nodeId,
|
||||||
@@ -287,11 +292,39 @@ function quizEvent(
|
|||||||
ref: part.toolCallId,
|
ref: part.toolCallId,
|
||||||
};
|
};
|
||||||
if (typeof answer.note === "string" && answer.note.length > 0) event.note = answer.note;
|
if (typeof answer.note === "string" && answer.note.length > 0) event.note = answer.note;
|
||||||
// The question text is the attribution input for untagged/unmapped probes.
|
// The question text is the attribution input for unattributed probes.
|
||||||
if (nodeId === null && typeof answer.questionText === "string") event.question = answer.questionText;
|
if (nodeId === null && typeof answer.questionText === "string") event.question = answer.questionText;
|
||||||
return event;
|
return event;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ─── the trigger helper (v1: every N quiz answers — swappable seam) ──────────
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How many ANSWERED quiz parts in this chat are not yet in the ledger — i.e.
|
||||||
|
* new evidence since the last extraction. The v1 trigger (turn-service)
|
||||||
|
* fires the pipeline when this crosses EXTRACTION.everyNQuizAnswers. Every
|
||||||
|
* answered quiz always becomes an event (attribution only decides nodeId),
|
||||||
|
* so ref-not-in-ledger is a correct "not yet processed" marker.
|
||||||
|
*/
|
||||||
|
export function pendingQuizAnswers(chatId: string, deps: GraphDeps): number {
|
||||||
|
const record = deps.chatStore.getChat(chatId);
|
||||||
|
if (!record) return 0;
|
||||||
|
const seenRefs = new Set<string>();
|
||||||
|
for (const event of deps.graphStore.readEvents()) {
|
||||||
|
if (event.chatId === chatId && event.ref) seenRefs.add(event.ref);
|
||||||
|
}
|
||||||
|
let count = 0;
|
||||||
|
for (const part of toolParts(record.messages)) {
|
||||||
|
if (part.name !== "quiz" || part.state !== "output-available") continue;
|
||||||
|
const out = part.output;
|
||||||
|
if (typeof out !== "object" || out === null) continue;
|
||||||
|
const answer = out as Partial<QuestionAnswerData>;
|
||||||
|
if (answer.kind !== "quiz" || answer.dismissed === true) continue; // dismissed = not answered
|
||||||
|
if (!seenRefs.has(part.toolCallId)) count++;
|
||||||
|
}
|
||||||
|
return count;
|
||||||
|
}
|
||||||
|
|
||||||
// ─── derivation (reads only — everything below is recomputable) ─────────────
|
// ─── derivation (reads only — everything below is recomputable) ─────────────
|
||||||
|
|
||||||
export interface DerivedGraph {
|
export interface DerivedGraph {
|
||||||
@@ -417,33 +450,3 @@ export function graphSnapshot(deps: GraphDeps): GraphSnapshot {
|
|||||||
|
|
||||||
return { nodes, edges, nodeStates, sessions, nodeSessions };
|
return { nodes, edges, nodeStates, sessions, nodeSessions };
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── catch-up ────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Fold every chat whose updatedAt is newer than its checkpoint (partial
|
|
||||||
* sessions deposit value — vision §1 "frequent updates"). Called at the top
|
|
||||||
* of GET /api/graph; the chat-handler onFinish hook covers the live path.
|
|
||||||
*/
|
|
||||||
export function foldStaleChats(deps: GraphDeps): number {
|
|
||||||
const checkpoints = new Map<string, string>();
|
|
||||||
for (const checkpoint of deps.graphStore.readCheckpoints()) {
|
|
||||||
const previous = checkpoints.get(checkpoint.chatId);
|
|
||||||
if (previous === undefined || checkpoint.foldedAt > previous) {
|
|
||||||
checkpoints.set(checkpoint.chatId, checkpoint.foldedAt);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
let folded = 0;
|
|
||||||
for (const chat of deps.chatStore.listChats()) {
|
|
||||||
const foldedAt = checkpoints.get(chat.id);
|
|
||||||
if (foldedAt !== undefined && chat.updatedAt <= foldedAt) continue;
|
|
||||||
try {
|
|
||||||
emitChat(chat.id, deps);
|
|
||||||
folded++;
|
|
||||||
} catch (err) {
|
|
||||||
console.error(`[graph] fold failed for chat ${chat.id}:`, err);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return folded;
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,9 +1,10 @@
|
|||||||
import { createAgentUIStreamResponse, type UIMessage } from "ai";
|
import { createAgentUIStreamResponse, type UIMessage } from "ai";
|
||||||
import { createTeacherAgent, type TeacherAgent } from "../agent/teacher-agent.ts";
|
import { createTeacherAgent, type TeacherAgent } from "../agent/teacher-agent.ts";
|
||||||
import { createJobRunner } from "../agent/subagents/runner.ts";
|
import { createJobRunner } from "../agent/subagents/runner.ts";
|
||||||
|
import { createExtractionPipeline } from "../agent/subagents/extraction.ts";
|
||||||
import { createTeacherTools } from "../agent/tools.ts";
|
import { createTeacherTools } from "../agent/tools.ts";
|
||||||
import { stores } from "./stores.ts";
|
import { stores } from "./stores.ts";
|
||||||
import { buildTeacherContext, foldTurnIntoGraph, persistTurn, spawnTurnJobs } from "./turn-service.ts";
|
import { buildTeacherContext, persistTurn, spawnTurnJobs, triggerExtraction } from "./turn-service.ts";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* One chat turn: run the teacher agent over the client's message list and
|
* One chat turn: run the teacher agent over the client's message list and
|
||||||
@@ -18,6 +19,7 @@ import { buildTeacherContext, foldTurnIntoGraph, persistTurn, spawnTurnJobs } fr
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
const jobRunner = createJobRunner(stores);
|
const jobRunner = createJobRunner(stores);
|
||||||
|
const extraction = createExtractionPipeline(stores);
|
||||||
|
|
||||||
export async function handleChatRequest(req: Request, agent?: TeacherAgent): Promise<Response> {
|
export async function handleChatRequest(req: Request, agent?: TeacherAgent): Promise<Response> {
|
||||||
let body: { chatId?: unknown; messages?: unknown };
|
let body: { chatId?: unknown; messages?: unknown };
|
||||||
@@ -46,7 +48,7 @@ export async function handleChatRequest(req: Request, agent?: TeacherAgent): Pro
|
|||||||
onFinish: ({ messages, isAborted }) => {
|
onFinish: ({ messages, isAborted }) => {
|
||||||
persistTurn(stores.chatStore, chatId, messages as UIMessage[], isAborted);
|
persistTurn(stores.chatStore, chatId, messages as UIMessage[], isAborted);
|
||||||
spawnTurnJobs(jobRunner, chatId, messages as UIMessage[]);
|
spawnTurnJobs(jobRunner, chatId, messages as UIMessage[]);
|
||||||
foldTurnIntoGraph(stores, chatId);
|
triggerExtraction(stores, extraction, chatId);
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|||||||
+5
-7
@@ -1,15 +1,13 @@
|
|||||||
import type { GraphSnapshot } from "../core/contracts.ts";
|
import type { GraphSnapshot } from "../core/contracts.ts";
|
||||||
import { foldStaleChats, graphSnapshot } from "../core/graph.ts";
|
import { graphSnapshot } from "../core/graph.ts";
|
||||||
import { stores } from "./stores.ts";
|
import { stores } from "./stores.ts";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Server-side graph entry for the home view (GET /api/graph): fold any chat
|
* Server-side graph entry for the home view (GET /api/graph). A PURE READ:
|
||||||
* whose updatedAt outstrips its emission checkpoint first (partial sessions
|
* the extraction pipeline (the sole graph writer) is triggered by quiz
|
||||||
* deposit value), then return the snapshot. Today the fold is the naive-union
|
* answers in sessions (turn-service), never from here — the home view
|
||||||
* emission; the extraction-pipeline stage replaces the writer behind this
|
* reflects whatever the pipeline has merged so far.
|
||||||
* same read surface.
|
|
||||||
*/
|
*/
|
||||||
export function graphView(): GraphSnapshot {
|
export function graphView(): GraphSnapshot {
|
||||||
foldStaleChats(stores);
|
|
||||||
return graphSnapshot(stores);
|
return graphSnapshot(stores);
|
||||||
}
|
}
|
||||||
|
|||||||
+29
-12
@@ -7,10 +7,11 @@
|
|||||||
* buildTeacherContext — graph → instructions block (the W8 read channel)
|
* buildTeacherContext — graph → instructions block (the W8 read channel)
|
||||||
* persistTurn — the canonical state write (client + agent messages)
|
* persistTurn — the canonical state write (client + agent messages)
|
||||||
* spawnTurnJobs — the async sub-agent protocol kick-off
|
* spawnTurnJobs — the async sub-agent protocol kick-off
|
||||||
* foldTurnIntoGraph — session → graph consolidation (the write channel;
|
* triggerExtraction — the graph write channel: fires the extraction
|
||||||
* the extraction-pipeline stage swaps this body for
|
* pipeline (the sole graph writer) once a chat
|
||||||
* the extractor/auditor/merger trigger behind the
|
* crosses the v1 quiz-answer threshold. The trigger
|
||||||
* same seam)
|
* mechanism is a v1 placeholder — this seam is the
|
||||||
|
* ONLY thing to swap when a better one lands.
|
||||||
*
|
*
|
||||||
* Logs are the minimal-observability baseline: one line per failure, tagged
|
* Logs are the minimal-observability baseline: one line per failure, tagged
|
||||||
* [chat <id>] — structured logging lands behind these seams later.
|
* [chat <id>] — structured logging lands behind these seams later.
|
||||||
@@ -18,9 +19,11 @@
|
|||||||
|
|
||||||
import type { UIMessage } from "ai";
|
import type { UIMessage } from "ai";
|
||||||
import { buildGraphContext } from "../core/graph-context.ts";
|
import { buildGraphContext } from "../core/graph-context.ts";
|
||||||
import { emitChat, getGraph, type GraphDeps } from "../core/graph.ts";
|
import { getGraph, pendingQuizAnswers } from "../core/graph.ts";
|
||||||
import type { ChatStore, GraphStore } from "../core/ports.ts";
|
import type { ChatStore, GraphStore } from "../core/ports.ts";
|
||||||
|
import { EXTRACTION } from "../../config.ts";
|
||||||
import type { JobRunner } from "../agent/subagents/runner.ts";
|
import type { JobRunner } from "../agent/subagents/runner.ts";
|
||||||
|
import type { ExtractionPipeline } from "../agent/subagents/extraction.ts";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* W8 (vision §1 "the graph as session context"): the learner's knowledge
|
* W8 (vision §1 "the graph as session context"): the learner's knowledge
|
||||||
@@ -66,16 +69,30 @@ export function spawnTurnJobs(jobRunner: JobRunner, chatId: string, messages: UI
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* W4: fold this chat into the knowledge-graph ledgers (tree nodes / edges,
|
* The graph write channel (vision §1 construction): the extraction pipeline
|
||||||
* quiz outcomes, taught events) — synchronous but tiny (append-only JSONL),
|
* (extractor → auditor → merger — the sole graph writer) runs ASYNC once a
|
||||||
* and errors never touch the response. Skipped under tests: scripted chats
|
* chat accumulates EXTRACTION.everyNQuizAnswers new answered quizzes since
|
||||||
* are fixtures, not knowledge.
|
* its last checkpoint. Fire-and-forget — never blocks the turn; failures
|
||||||
|
* are logged by the pipeline and leave the checkpoint untouched, so the
|
||||||
|
* next trigger re-runs the whole thing.
|
||||||
|
*
|
||||||
|
* v1 placeholder trigger, deliberately isolated in this one seam: swap
|
||||||
|
* here when a better mechanism lands — the pipeline is trigger-agnostic.
|
||||||
|
* Skipped under tests: scripted chats are fixtures, not knowledge (the
|
||||||
|
* pipeline itself is tested directly with mock models).
|
||||||
*/
|
*/
|
||||||
export function foldTurnIntoGraph(deps: GraphDeps, chatId: string): void {
|
export function triggerExtraction(
|
||||||
|
deps: { chatStore: ChatStore; graphStore: GraphStore },
|
||||||
|
extraction: ExtractionPipeline,
|
||||||
|
chatId: string,
|
||||||
|
): void {
|
||||||
if (process.env.NODE_ENV === "test") return;
|
if (process.env.NODE_ENV === "test") return;
|
||||||
try {
|
try {
|
||||||
emitChat(chatId, deps);
|
if (pendingQuizAnswers(chatId, deps) < EXTRACTION.everyNQuizAnswers) return;
|
||||||
|
void extraction.runExtraction(chatId).catch((err) => {
|
||||||
|
console.error(`[chat ${chatId}] extraction pipeline failed:`, err);
|
||||||
|
});
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
console.error(`[chat ${chatId}] graph emission failed:`, err);
|
console.error(`[chat ${chatId}] extraction trigger failed:`, err);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+3
-3
@@ -6,8 +6,8 @@
|
|||||||
* session asserting the same (from,to,type)
|
* session asserting the same (from,to,type)
|
||||||
* appends another line; reads merge)
|
* appends another line; reads merge)
|
||||||
* events.jsonl LedgerEvent — quiz outcomes + taught events
|
* events.jsonl LedgerEvent — quiz outcomes + taught events
|
||||||
* mappings.jsonl NodeMapping — the naive-union identity map:
|
* mappings.jsonl NodeMapping — the session→graph identity map:
|
||||||
* (chatId, treeNodeId) → graphNodeId
|
* (chatId, sourceId) → graphNodeId
|
||||||
* sessions.jsonl SessionCheckpoint— emission checkpoints (foldedAt)
|
* sessions.jsonl SessionCheckpoint— emission checkpoints (foldedAt)
|
||||||
*
|
*
|
||||||
* Implements core/ports.ts GraphStore. data/graph/* is exclusively this
|
* Implements core/ports.ts GraphStore. data/graph/* is exclusively this
|
||||||
@@ -105,7 +105,7 @@ const isLedgerEvent = (v: unknown): v is LedgerEvent =>
|
|||||||
isString(v.at);
|
isString(v.at);
|
||||||
|
|
||||||
const isNodeMapping = (v: unknown): v is NodeMapping =>
|
const isNodeMapping = (v: unknown): v is NodeMapping =>
|
||||||
isRecord(v) && isString(v.chatId) && isString(v.treeNodeId) && isString(v.graphNodeId);
|
isRecord(v) && isString(v.chatId) && isString(v.sourceId) && isString(v.graphNodeId);
|
||||||
|
|
||||||
const isCheckpoint = (v: unknown): v is SessionCheckpoint =>
|
const isCheckpoint = (v: unknown): v is SessionCheckpoint =>
|
||||||
isRecord(v) && isString(v.chatId) && isString(v.foldedAt);
|
isRecord(v) && isString(v.chatId) && isString(v.foldedAt);
|
||||||
|
|||||||
@@ -0,0 +1,373 @@
|
|||||||
|
/**
|
||||||
|
* The extraction pipeline tests (vision §1 construction): extractor →
|
||||||
|
* auditor → merger, the sole graph writer.
|
||||||
|
*
|
||||||
|
* transcriptView — the compact view both LLM stages see (pure)
|
||||||
|
* pipeline — MockLanguageModelV4 drives both agents: findings out,
|
||||||
|
* audit verdict in, deterministic merge applied; failure
|
||||||
|
* paths checkpoint nothing; single-flight per chat
|
||||||
|
*
|
||||||
|
* The merger itself (identity resolution, event rebuild, idempotency) is
|
||||||
|
* tested in tests/graph.test.ts directly against mergeFindings — here it is
|
||||||
|
* the integration: what one full pipeline run does to the ledgers.
|
||||||
|
*
|
||||||
|
* Ledger: in-memory (the GraphStore port) — this suite never touches
|
||||||
|
* data/graph. Chats: real ChatRecords in the real chat store (created and
|
||||||
|
* deleted by this suite).
|
||||||
|
*/
|
||||||
|
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
import * as fs from "node:fs";
|
||||||
|
import { after, describe, it } from "node:test";
|
||||||
|
import { MockLanguageModelV4 } from "ai/test";
|
||||||
|
import type { UIMessage } from "ai";
|
||||||
|
import type { GraphEdge, GraphNode, LedgerEvent, NodeMapping, SessionCheckpoint, SessionFindings } from "../src/core/contracts.ts";
|
||||||
|
import type { GraphStore } from "../src/core/ports.ts";
|
||||||
|
import { createTree, markDone } from "../src/core/plan-tree.ts";
|
||||||
|
import { createExtractionPipeline, transcriptView } from "../src/agent/subagents/extraction.ts";
|
||||||
|
import { chatStore } from "../src/store/chats.ts";
|
||||||
|
import { PATHS } from "../config.ts";
|
||||||
|
|
||||||
|
// ─── fixtures ────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
const createdChats: string[] = [];
|
||||||
|
|
||||||
|
function usage() {
|
||||||
|
return {
|
||||||
|
inputTokens: { total: 10, noCache: 10, cacheRead: 0, cacheWrite: 0 },
|
||||||
|
outputTokens: { total: 5, text: 5, reasoning: 0 },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** An in-memory GraphStore (core/ports.ts) — this suite never touches data/graph. */
|
||||||
|
function memoryLedgers() {
|
||||||
|
const nodes: GraphNode[] = [];
|
||||||
|
const edges: GraphEdge[] = [];
|
||||||
|
const events: LedgerEvent[] = [];
|
||||||
|
const mappings: NodeMapping[] = [];
|
||||||
|
const checkpoints: SessionCheckpoint[] = [];
|
||||||
|
return {
|
||||||
|
appendNode: (n: GraphNode) => void nodes.push(n),
|
||||||
|
readNodes: () => nodes.map((n) => ({ ...n })),
|
||||||
|
appendEdge: (e: GraphEdge) => void edges.push(e),
|
||||||
|
readEdges: () => edges.map((e) => ({ ...e })),
|
||||||
|
appendEvent: (e: LedgerEvent) => void events.push(e),
|
||||||
|
readEvents: () => events.map((e) => ({ ...e })),
|
||||||
|
appendMapping: (m: NodeMapping) => void mappings.push(m),
|
||||||
|
readMappings: () => mappings.map((m) => ({ ...m })),
|
||||||
|
appendCheckpoint: (c: SessionCheckpoint) => void checkpoints.push(c),
|
||||||
|
readCheckpoints: () => checkpoints.map((c) => ({ ...c })),
|
||||||
|
} satisfies GraphStore;
|
||||||
|
}
|
||||||
|
|
||||||
|
function assistant(id: string, parts: unknown[]): UIMessage {
|
||||||
|
return { id, role: "assistant", parts } as unknown as UIMessage;
|
||||||
|
}
|
||||||
|
|
||||||
|
function user(id: string, text: string): UIMessage {
|
||||||
|
return { id, role: "user", parts: [{ type: "text", text }] } as unknown as UIMessage;
|
||||||
|
}
|
||||||
|
|
||||||
|
function quizPart(toolCallId: string, input: Record<string, unknown>, output: Record<string, unknown>): unknown {
|
||||||
|
return { type: "tool-quiz", toolCallId, state: "output-available", input, output };
|
||||||
|
}
|
||||||
|
|
||||||
|
function planPart(toolCallId: string, name: string, input: Record<string, unknown>, tree: ReturnType<typeof createTree>): unknown {
|
||||||
|
return {
|
||||||
|
type: `tool-${name}`,
|
||||||
|
toolCallId,
|
||||||
|
state: "output-available",
|
||||||
|
input,
|
||||||
|
output: { ok: true, tree },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A session-shaped fixture chat: probe → plan → check → mark_done. */
|
||||||
|
function fixtureChat() {
|
||||||
|
const tree = markDone(
|
||||||
|
createTree(
|
||||||
|
[
|
||||||
|
{ id: "cr", title: "Chain rule intuition" },
|
||||||
|
{ id: "apply", title: "Applying the chain rule" },
|
||||||
|
],
|
||||||
|
[{ from: "cr", to: "apply" }],
|
||||||
|
),
|
||||||
|
"cr",
|
||||||
|
);
|
||||||
|
const chat = chatStore.createChat("Extraction fixture");
|
||||||
|
createdChats.push(chat.id);
|
||||||
|
chatStore.saveTree(chat.id, tree);
|
||||||
|
chatStore.saveMessages(chat.id, [
|
||||||
|
user("u1", "teach me the chain rule"),
|
||||||
|
assistant("a1", [
|
||||||
|
quizPart("probe-1", { question: "What is a derivative, roughly?" }, {
|
||||||
|
questionId: "probe-1", kind: "quiz", questionText: "What is a derivative, roughly?",
|
||||||
|
correct: false, dontKnow: false, answeredAt: "2026-08-22T01:00:00.000Z",
|
||||||
|
}),
|
||||||
|
]),
|
||||||
|
assistant("a2", [
|
||||||
|
{ type: "text", text: "The derivative measures how fast a function changes." },
|
||||||
|
planPart("p1", "plan_init", { nodes: [], edges: [] }, tree),
|
||||||
|
]),
|
||||||
|
assistant("a3", [
|
||||||
|
quizPart("check-1", { question: "Check", nodeId: "apply" }, {
|
||||||
|
questionId: "check-1", kind: "quiz", questionText: "Check", correct: true, dontKnow: false,
|
||||||
|
answeredAt: "2026-08-22T02:00:00.000Z",
|
||||||
|
}),
|
||||||
|
]),
|
||||||
|
assistant("a4", [planPart("p2", "plan_mark_done", { id: "cr" }, tree)]),
|
||||||
|
]);
|
||||||
|
return { chatId: chat.id, record: chatStore.getChat(chat.id)! };
|
||||||
|
}
|
||||||
|
|
||||||
|
after(() => {
|
||||||
|
for (const id of createdChats) {
|
||||||
|
try {
|
||||||
|
fs.unlinkSync(`${PATHS.chats}/${id}.json`);
|
||||||
|
} catch {
|
||||||
|
/* already gone */
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
/** What the extractor would produce for the fixture session. */
|
||||||
|
const FINDINGS: SessionFindings = {
|
||||||
|
propositions: [
|
||||||
|
{
|
||||||
|
id: "p1",
|
||||||
|
statement: "Chain rule intuition",
|
||||||
|
aliases: [],
|
||||||
|
kind: "truth",
|
||||||
|
treeNodeId: "cr",
|
||||||
|
evidence: ["TEACHER: The derivative measures how fast a function changes."],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "p2",
|
||||||
|
statement: "Applying the chain rule",
|
||||||
|
aliases: [],
|
||||||
|
kind: "truth",
|
||||||
|
treeNodeId: "apply",
|
||||||
|
evidence: ["FINAL PLAN TREE (node ids): cr = Chain rule intuition; apply = Applying the chain rule"],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
edges: [{ from: "p1", to: "p2", type: "requires", evidence: ["TREE EDGES: cr requires apply"] }],
|
||||||
|
attribution: { quiz: { "probe-1": "p1" } },
|
||||||
|
};
|
||||||
|
|
||||||
|
/** A model whose ONE tool call is `name` with the given input; then text. */
|
||||||
|
function submitModel(name: string, input: unknown): MockLanguageModelV4 {
|
||||||
|
let turn = 0;
|
||||||
|
return new MockLanguageModelV4({
|
||||||
|
doGenerate: async () => {
|
||||||
|
turn++;
|
||||||
|
if (turn === 1) {
|
||||||
|
return {
|
||||||
|
content: [
|
||||||
|
{
|
||||||
|
type: "tool-call" as const,
|
||||||
|
toolCallId: `${name}-call`,
|
||||||
|
toolName: name,
|
||||||
|
input: JSON.stringify(input),
|
||||||
|
},
|
||||||
|
],
|
||||||
|
finishReason: { unified: "tool-calls" as const, raw: "tool-calls" },
|
||||||
|
usage: usage(),
|
||||||
|
warnings: [],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
content: [{ type: "text" as const, text: "submitted" }],
|
||||||
|
finishReason: { unified: "stop" as const, raw: "stop" },
|
||||||
|
usage: usage(),
|
||||||
|
warnings: [],
|
||||||
|
};
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A model that never calls a tool (the failure path). */
|
||||||
|
function silentModel(): MockLanguageModelV4 {
|
||||||
|
return new MockLanguageModelV4({
|
||||||
|
doGenerate: async () => ({
|
||||||
|
content: [{ type: "text" as const, text: "I refuse to use tools" }],
|
||||||
|
finishReason: { unified: "stop" as const, raw: "stop" },
|
||||||
|
usage: usage(),
|
||||||
|
warnings: [],
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── transcriptView (pure) ───────────────────────────────────────────────────
|
||||||
|
|
||||||
|
describe("transcriptView (pure)", () => {
|
||||||
|
it("shows the final tree, verbatim conversation, structured events — plan ops folded", () => {
|
||||||
|
const { record } = fixtureChat();
|
||||||
|
const view = transcriptView(record);
|
||||||
|
|
||||||
|
// The final tree with node ids (the extractor's backbone + the
|
||||||
|
// pruned-out stays pruned — only the FINAL tree is shown).
|
||||||
|
assert.match(view, /SESSION: Extraction fixture/);
|
||||||
|
assert.match(view, /FINAL PLAN TREE \(node ids\): cr = Chain rule intuition; apply = Applying the chain rule/);
|
||||||
|
assert.match(view, /TREE EDGES: cr requires apply/);
|
||||||
|
|
||||||
|
// Conversation verbatim — evidence quotes can come from it.
|
||||||
|
assert.match(view, /LEARNER: teach me the chain rule/);
|
||||||
|
assert.match(view, /TEACHER: The derivative measures how fast a function changes\./);
|
||||||
|
|
||||||
|
// Structured events, attribution-ready.
|
||||||
|
assert.match(view, /QUIZ \[probe-1\] What is a derivative, roughly\? — the learner answered incorrectly/);
|
||||||
|
assert.match(view, /QUIZ \[check-1\] Check — the learner answered correctly/);
|
||||||
|
assert.match(view, /TEACHER marked tree node done \[p2\]: cr/);
|
||||||
|
|
||||||
|
// plan_init / plan_mark_done tool parts themselves are folded into
|
||||||
|
// the tree line — no raw JSON noise.
|
||||||
|
assert.doesNotMatch(view, /plan_init/);
|
||||||
|
assert.doesNotMatch(view, /output-available/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("summarizes job tool results as one line and drops empty text", () => {
|
||||||
|
const chat = chatStore.createChat("View noise fixture");
|
||||||
|
createdChats.push(chat.id);
|
||||||
|
chatStore.saveMessages(chat.id, [
|
||||||
|
user("u1", "go"),
|
||||||
|
assistant("a1", [
|
||||||
|
{
|
||||||
|
type: "tool-research",
|
||||||
|
toolCallId: "r1",
|
||||||
|
state: "output-available",
|
||||||
|
input: { question: "q" },
|
||||||
|
output: { brief: "long brief", sources: [] },
|
||||||
|
},
|
||||||
|
{ type: "text", text: " " },
|
||||||
|
]),
|
||||||
|
]);
|
||||||
|
const view = transcriptView(chatStore.getChat(chat.id)!);
|
||||||
|
assert.match(view, /\(research tool result delivered\)/);
|
||||||
|
assert.doesNotMatch(view, /long brief/);
|
||||||
|
assert.doesNotMatch(view, /TEACHER: {2}/);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ─── the pipeline ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
describe("the extraction pipeline (extractor → auditor → merger)", () => {
|
||||||
|
it("runs end to end: findings → audit verdict → deterministic merge + checkpoint", async () => {
|
||||||
|
const { chatId } = fixtureChat();
|
||||||
|
const ledgers = memoryLedgers();
|
||||||
|
|
||||||
|
// The auditor passes p1 through, drops p2 (unbacked), re-points the
|
||||||
|
// probe attribution to p1, and records the drop.
|
||||||
|
const verified: SessionFindings = {
|
||||||
|
propositions: [FINDINGS.propositions[0]!],
|
||||||
|
edges: [],
|
||||||
|
attribution: { quiz: { "probe-1": "p1" } },
|
||||||
|
};
|
||||||
|
|
||||||
|
const pipeline = createExtractionPipeline({
|
||||||
|
chatStore,
|
||||||
|
graphStore: ledgers,
|
||||||
|
extractor: submitModel("submit_findings", FINDINGS),
|
||||||
|
auditor: submitModel("submit_review", { ...verified, dropped: [{ ref: "p2", reason: "evidence not found" }] }),
|
||||||
|
});
|
||||||
|
const stats = await pipeline.runExtraction(chatId);
|
||||||
|
|
||||||
|
assert.deepEqual(stats, { nodes: 1, edges: 0, events: 3 });
|
||||||
|
|
||||||
|
// Only the audited proposition became a node; mapping provenance kept.
|
||||||
|
assert.equal(ledgers.readNodes().length, 1);
|
||||||
|
assert.equal(ledgers.readNodes()[0]!.statement, "Chain rule intuition");
|
||||||
|
assert.deepEqual(ledgers.readMappings(), [{ chatId, sourceId: "cr", graphNodeId: ledgers.readNodes()[0]!.id }]);
|
||||||
|
|
||||||
|
// Events rebuilt deterministically from the transcript — the probe
|
||||||
|
// retroactively attributed, the check tagged, mark_done mapped.
|
||||||
|
const events = ledgers.readEvents();
|
||||||
|
assert.deepEqual(
|
||||||
|
events.map((e) => [e.kind, e.ref]),
|
||||||
|
[
|
||||||
|
["probe-miss", "probe-1"],
|
||||||
|
["check-correct", "check-1"],
|
||||||
|
["taught", "taught:cr"],
|
||||||
|
],
|
||||||
|
);
|
||||||
|
const nodeId = ledgers.readNodes()[0]!.id;
|
||||||
|
assert.equal(events[0]!.nodeId, nodeId, "probe attributed via audit");
|
||||||
|
assert.equal(events[1]!.nodeId, null, "tagged to a dropped proposition → null");
|
||||||
|
assert.equal(events[2]!.nodeId, nodeId, "taught via tree-node mapping");
|
||||||
|
|
||||||
|
// Checkpointed through the record's updatedAt — never the same turns twice.
|
||||||
|
assert.equal(ledgers.readCheckpoints().length, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("checkpoints nothing when a stage fails", async () => {
|
||||||
|
const { chatId } = fixtureChat();
|
||||||
|
const ledgers = memoryLedgers();
|
||||||
|
|
||||||
|
const pipeline = createExtractionPipeline({
|
||||||
|
chatStore,
|
||||||
|
graphStore: ledgers,
|
||||||
|
extractor: submitModel("submit_findings", FINDINGS),
|
||||||
|
auditor: silentModel(), // never submits the review
|
||||||
|
});
|
||||||
|
await assert.rejects(pipeline.runExtraction(chatId), /the auditor never submitted a review/);
|
||||||
|
assert.equal(ledgers.readNodes().length, 0);
|
||||||
|
assert.equal(ledgers.readCheckpoints().length, 0, "no checkpoint — the next trigger re-runs");
|
||||||
|
|
||||||
|
const failingExtractor = createExtractionPipeline({
|
||||||
|
chatStore,
|
||||||
|
graphStore: ledgers,
|
||||||
|
extractor: silentModel(),
|
||||||
|
auditor: submitModel("submit_review", FINDINGS),
|
||||||
|
});
|
||||||
|
await assert.rejects(failingExtractor.runExtraction(chatId), /the extractor never submitted findings/);
|
||||||
|
assert.equal(ledgers.readCheckpoints().length, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("is single-flight per chat — a concurrent second run is a no-op", async () => {
|
||||||
|
const { chatId } = fixtureChat();
|
||||||
|
const ledgers = memoryLedgers();
|
||||||
|
|
||||||
|
// The extractor's first call hangs until we release it.
|
||||||
|
let release!: () => void;
|
||||||
|
const gate = new Promise<void>((resolve) => {
|
||||||
|
release = resolve;
|
||||||
|
});
|
||||||
|
let calls = 0;
|
||||||
|
const gatedExtractor = new MockLanguageModelV4({
|
||||||
|
doGenerate: async () => {
|
||||||
|
calls++;
|
||||||
|
if (calls === 1) await gate;
|
||||||
|
return {
|
||||||
|
content: [{ type: "text" as const, text: "…" }],
|
||||||
|
finishReason: { unified: "stop" as const, raw: "stop" },
|
||||||
|
usage: usage(),
|
||||||
|
warnings: [],
|
||||||
|
};
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
const pipeline = createExtractionPipeline({
|
||||||
|
chatStore,
|
||||||
|
graphStore: ledgers,
|
||||||
|
extractor: gatedExtractor,
|
||||||
|
auditor: silentModel(),
|
||||||
|
});
|
||||||
|
|
||||||
|
const first = pipeline.runExtraction(chatId);
|
||||||
|
const second = await pipeline.runExtraction(chatId); // in-flight → zeros
|
||||||
|
assert.deepEqual(second, { nodes: 0, edges: 0, events: 0 });
|
||||||
|
|
||||||
|
release();
|
||||||
|
await assert.rejects(first, /the extractor never submitted findings/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("throws for unknown chats", async () => {
|
||||||
|
const ledgers = memoryLedgers();
|
||||||
|
const pipeline = createExtractionPipeline({
|
||||||
|
chatStore,
|
||||||
|
graphStore: ledgers,
|
||||||
|
extractor: silentModel(),
|
||||||
|
auditor: silentModel(),
|
||||||
|
});
|
||||||
|
await assert.rejects(pipeline.runExtraction("chat_does_not_exist"), /Chat not found/);
|
||||||
|
});
|
||||||
|
});
|
||||||
+71
-47
@@ -5,10 +5,11 @@
|
|||||||
* buildGraphContext — per-request entry point (derived-graph → block)
|
* buildGraphContext — per-request entry point (derived-graph → block)
|
||||||
* injection plumbing — chat-flow style: the teacher's instructions carry
|
* injection plumbing — chat-flow style: the teacher's instructions carry
|
||||||
* the block (MockLanguageModelV4 captures the prompt)
|
* the block (MockLanguageModelV4 captures the prompt)
|
||||||
* exact-match merge — emitChat dedupes case/whitespace-identical titles
|
* exact-match merge — the merger (mergeFindings) dedupes
|
||||||
* onto existing nodes (no fuzzy matching)
|
* case/whitespace-identical statements onto existing
|
||||||
|
* nodes (no fuzzy matching)
|
||||||
*
|
*
|
||||||
* Ledger isolation: NONE needed for the merge tests — emitChat's ledger is
|
* Ledger isolation: NONE needed for the merge tests — the merger's ledger is
|
||||||
* injectable (in-memory), so the live data/graph files are only ever READ
|
* injectable (in-memory), so the live data/graph files are only ever READ
|
||||||
* here (tests/graph.test.ts owns the backup-and-clear dance; both suites run
|
* here (tests/graph.test.ts owns the backup-and-clear dance; both suites run
|
||||||
* concurrently as separate processes). Chat fixtures are real ChatRecords,
|
* concurrently as separate processes). Chat fixtures are real ChatRecords,
|
||||||
@@ -21,9 +22,9 @@ import * as path from "node:path";
|
|||||||
import { after, describe, it } from "node:test";
|
import { after, describe, it } from "node:test";
|
||||||
import { MockLanguageModelV4, simulateReadableStream } from "ai/test";
|
import { MockLanguageModelV4, simulateReadableStream } from "ai/test";
|
||||||
import type { LanguageModelV4StreamPart } from "@ai-sdk/provider";
|
import type { LanguageModelV4StreamPart } from "@ai-sdk/provider";
|
||||||
import type { GraphEdge, GraphNode, LedgerEvent } from "../src/core/contracts.ts";
|
import type { GraphEdge, GraphNode, LedgerEvent, SessionFindings } from "../src/core/contracts.ts";
|
||||||
import type { NodeStateClass } from "../src/core/contracts.ts";
|
import type { NodeStateClass } from "../src/core/contracts.ts";
|
||||||
import { emitChat, getGraph } from "../src/core/graph.ts";
|
import { getGraph, mergeFindings } from "../src/core/graph.ts";
|
||||||
import type { GraphStore } from "../src/core/ports.ts";
|
import type { GraphStore } from "../src/core/ports.ts";
|
||||||
import {
|
import {
|
||||||
buildGraphContext,
|
buildGraphContext,
|
||||||
@@ -295,7 +296,7 @@ describe("graph-context injection (chat-flow style)", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// ─── exact-match merge (emitChat dedup) ─────────────────────────────────────
|
// ─── exact-match merge (merger identity resolution) ──────────────────────────
|
||||||
|
|
||||||
/** An in-memory stand-in for the five ledgers (the GraphStore port, core/ports.ts). */
|
/** An in-memory stand-in for the five ledgers (the GraphStore port, core/ports.ts). */
|
||||||
function memoryLedgers() {
|
function memoryLedgers() {
|
||||||
@@ -320,7 +321,31 @@ function memoryLedgers() {
|
|||||||
} satisfies GraphStore & { counts: { nodes: number; mappings: number } };
|
} satisfies GraphStore & { counts: { nodes: number; mappings: number } };
|
||||||
}
|
}
|
||||||
|
|
||||||
/** A real chat record with a stored tree (emitChat reads the real chat store). */
|
/** Findings as the extractor would file them for a plan tree: one
|
||||||
|
* proposition per node (statement = title, treeNodeId set), one `requires`
|
||||||
|
* edge per tree edge — the merge tests drive the merger with these. */
|
||||||
|
function treeFindings(tree: ReturnType<typeof createTree>): SessionFindings {
|
||||||
|
const idByNodeId = new Map(tree.nodes.map((n, i) => [n.id, `p${i + 1}`]));
|
||||||
|
return {
|
||||||
|
propositions: tree.nodes.map((n, i) => ({
|
||||||
|
id: `p${i + 1}`,
|
||||||
|
statement: n.title,
|
||||||
|
aliases: [],
|
||||||
|
kind: "truth" as const,
|
||||||
|
treeNodeId: n.id,
|
||||||
|
evidence: ["fixture evidence"],
|
||||||
|
})),
|
||||||
|
edges: tree.edges.map((e) => ({
|
||||||
|
from: idByNodeId.get(e.from)!,
|
||||||
|
to: idByNodeId.get(e.to)!,
|
||||||
|
type: "requires" as const,
|
||||||
|
evidence: ["fixture evidence"],
|
||||||
|
})),
|
||||||
|
attribution: { quiz: {} },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A real chat record with a stored tree (the merger reads the real chat store). */
|
||||||
function fixtureChat(title: string, tree: ReturnType<typeof createTree>) {
|
function fixtureChat(title: string, tree: ReturnType<typeof createTree>) {
|
||||||
const chat = chatStore.createChat(title);
|
const chat = chatStore.createChat(title);
|
||||||
createdChats.push(chat.id);
|
createdChats.push(chat.id);
|
||||||
@@ -329,24 +354,26 @@ function fixtureChat(title: string, tree: ReturnType<typeof createTree>) {
|
|||||||
return chatStore.getChat(chat.id)!;
|
return chatStore.getChat(chat.id)!;
|
||||||
}
|
}
|
||||||
|
|
||||||
describe("exact-match merge (emitChat)", () => {
|
describe("exact-match merge (merger identity resolution)", () => {
|
||||||
it("maps a case/whitespace-identical title onto the existing node instead of duplicating", () => {
|
it("maps a case/whitespace-identical title onto the existing node instead of duplicating", () => {
|
||||||
const ledgers = memoryLedgers();
|
const ledgers = memoryLedgers();
|
||||||
const chat1 = fixtureChat("Merge A", createTree([{ id: "cr", title: "Chain rule intuition" }], []));
|
const tree1 = createTree([{ id: "cr", title: "Chain rule intuition" }], []);
|
||||||
let stats = emitChat(chat1.id, { chatStore, graphStore: ledgers });
|
const chat1 = fixtureChat("Merge A", tree1);
|
||||||
|
let stats = mergeFindings(chat1.id, treeFindings(tree1), { chatStore, graphStore: ledgers });
|
||||||
assert.deepEqual(stats, { nodes: 1, edges: 0, events: 0 });
|
assert.deepEqual(stats, { nodes: 1, edges: 0, events: 0 });
|
||||||
const firstId = ledgers.readNodes()[0]!.id;
|
const firstId = ledgers.readNodes()[0]!.id;
|
||||||
|
|
||||||
// Second session, same proposition: different casing + padding.
|
// Second session, same proposition: different casing + padding.
|
||||||
const chat2 = fixtureChat("Merge B", createTree([{ id: "cr2", title: " chain Rule INTUITION " }], []));
|
const tree2 = createTree([{ id: "cr2", title: " chain Rule INTUITION " }], []);
|
||||||
stats = emitChat(chat2.id, { chatStore, graphStore: ledgers });
|
const chat2 = fixtureChat("Merge B", tree2);
|
||||||
|
stats = mergeFindings(chat2.id, treeFindings(tree2), { chatStore, graphStore: ledgers });
|
||||||
assert.deepEqual(stats, { nodes: 0, edges: 0, events: 0 }, "no duplicate node appended");
|
assert.deepEqual(stats, { nodes: 0, edges: 0, events: 0 }, "no duplicate node appended");
|
||||||
|
|
||||||
assert.equal(ledgers.counts.nodes, 1, "still exactly one graph node");
|
assert.equal(ledgers.counts.nodes, 1, "still exactly one graph node");
|
||||||
assert.equal(ledgers.readNodes()[0]!.id, firstId);
|
assert.equal(ledgers.readNodes()[0]!.id, firstId);
|
||||||
// Provenance stays honest: one mapping line per (chat, treeNodeId).
|
// Provenance stays honest: one mapping line per (chat, graph node).
|
||||||
assert.equal(ledgers.counts.mappings, 2);
|
assert.equal(ledgers.counts.mappings, 2);
|
||||||
assert.deepEqual(ledgers.readMappings()[1], { chatId: chat2.id, treeNodeId: "cr2", graphNodeId: firstId });
|
assert.deepEqual(ledgers.readMappings()[1], { chatId: chat2.id, sourceId: "cr2", graphNodeId: firstId });
|
||||||
});
|
});
|
||||||
|
|
||||||
it("matches aliases too (exact, case-insensitive)", () => {
|
it("matches aliases too (exact, case-insensitive)", () => {
|
||||||
@@ -358,52 +385,49 @@ describe("exact-match merge (emitChat)", () => {
|
|||||||
kind: "truth",
|
kind: "truth",
|
||||||
createdAt: "2026-08-22T00:00:00.000Z",
|
createdAt: "2026-08-22T00:00:00.000Z",
|
||||||
});
|
});
|
||||||
const chat = fixtureChat("Merge alias", createTree([{ id: "d", title: "The Derivative" }], []));
|
const tree = createTree([{ id: "d", title: "The Derivative" }], []);
|
||||||
const stats = emitChat(chat.id, { chatStore, graphStore: ledgers });
|
const chat = fixtureChat("Merge alias", tree);
|
||||||
|
const stats = mergeFindings(chat.id, treeFindings(tree), { chatStore, graphStore: ledgers });
|
||||||
assert.deepEqual(stats, { nodes: 0, edges: 0, events: 0 }, "alias match → no new node");
|
assert.deepEqual(stats, { nodes: 0, edges: 0, events: 0 }, "alias match → no new node");
|
||||||
assert.deepEqual(ledgers.readMappings(), [{ chatId: chat.id, treeNodeId: "d", graphNodeId: "n_alias" }]);
|
assert.deepEqual(ledgers.readMappings(), [{ chatId: chat.id, sourceId: "d", graphNodeId: "n_alias" }]);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("does NOT fuzzy-match near-identical titles (that stays deferred)", () => {
|
it("does NOT fuzzy-match near-identical titles (that stays deferred)", () => {
|
||||||
const ledgers = memoryLedgers();
|
const ledgers = memoryLedgers();
|
||||||
const chat1 = fixtureChat("Fuzzy A", createTree([{ id: "cr", title: "Chain rule intuition" }], []));
|
const tree1 = createTree([{ id: "cr", title: "Chain rule intuition" }], []);
|
||||||
emitChat(chat1.id, { chatStore, graphStore: ledgers });
|
const chat1 = fixtureChat("Fuzzy A", tree1);
|
||||||
const chat2 = fixtureChat(
|
mergeFindings(chat1.id, treeFindings(tree1), { chatStore, graphStore: ledgers });
|
||||||
"Fuzzy B",
|
const tree2 = createTree([{ id: "cr2", title: "Chain rules intuition" }], []);
|
||||||
createTree([{ id: "cr2", title: "Chain rules intuition" }], []),
|
const chat2 = fixtureChat("Fuzzy B", tree2);
|
||||||
);
|
const stats = mergeFindings(chat2.id, treeFindings(tree2), { chatStore, graphStore: ledgers });
|
||||||
const stats = emitChat(chat2.id, { chatStore, graphStore: ledgers });
|
assert.equal(stats.nodes, 1, "near-miss statements stay separate nodes");
|
||||||
assert.equal(stats.nodes, 1, "near-miss titles stay separate nodes");
|
|
||||||
assert.equal(ledgers.counts.nodes, 2);
|
assert.equal(ledgers.counts.nodes, 2);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("merged nodes unify the second chat's edges onto the existing graph node", () => {
|
it("merged nodes unify the second chat's edges onto the existing graph node", () => {
|
||||||
const ledgers = memoryLedgers();
|
const ledgers = memoryLedgers();
|
||||||
const chat1 = fixtureChat(
|
const tree1 = createTree(
|
||||||
"Edge A",
|
[
|
||||||
createTree(
|
{ id: "base", title: "Derivative as a limit" },
|
||||||
[
|
{ id: "chain", title: "Chain rule intuition" },
|
||||||
{ id: "base", title: "Derivative as a limit" },
|
],
|
||||||
{ id: "chain", title: "Chain rule intuition" },
|
[{ from: "base", to: "chain" }],
|
||||||
],
|
|
||||||
[{ from: "base", to: "chain" }],
|
|
||||||
),
|
|
||||||
);
|
);
|
||||||
emitChat(chat1.id, { chatStore, graphStore: ledgers });
|
const chat1 = fixtureChat("Edge A", tree1);
|
||||||
|
mergeFindings(chat1.id, treeFindings(tree1), { chatStore, graphStore: ledgers });
|
||||||
|
|
||||||
// Second session re-teaches the same two propositions (exact titles)
|
// Second session re-teaches the same two propositions (exact statements
|
||||||
// and asserts the same edge — it must dedupe onto the existing nodes.
|
// modulo case) and asserts the same edge — it must dedupe onto the
|
||||||
const chat2 = fixtureChat(
|
// existing nodes.
|
||||||
"Edge B",
|
const tree2 = createTree(
|
||||||
createTree(
|
[
|
||||||
[
|
{ id: "base", title: "derivative as a limit" },
|
||||||
{ id: "base", title: "derivative as a limit" },
|
{ id: "chain", title: "chain rule intuition" },
|
||||||
{ id: "chain", title: "chain rule intuition" },
|
],
|
||||||
],
|
[{ from: "base", to: "chain" }],
|
||||||
[{ from: "base", to: "chain" }],
|
|
||||||
),
|
|
||||||
);
|
);
|
||||||
const stats = emitChat(chat2.id, { chatStore, graphStore: ledgers });
|
const chat2 = fixtureChat("Edge B", tree2);
|
||||||
|
const stats = mergeFindings(chat2.id, treeFindings(tree2), { chatStore, graphStore: ledgers });
|
||||||
// A new ASSERTION line is appended (provenance is honest) — reads merge
|
// A new ASSERTION line is appended (provenance is honest) — reads merge
|
||||||
// by (from,to,type), so the derived graph still has one edge with both
|
// by (from,to,type), so the derived graph still has one edge with both
|
||||||
// sessions in its provenance.
|
// sessions in its provenance.
|
||||||
|
|||||||
+125
-87
@@ -1,21 +1,27 @@
|
|||||||
/**
|
/**
|
||||||
* Graph store + emission tests (W4; vision §1, contracts.ts "v1 emission").
|
* Graph merger + trigger + derivation tests (vision §1; contracts.ts
|
||||||
|
* "extraction pipeline"). The merger (core/graph.ts mergeFindings) is the
|
||||||
|
* pipeline's deterministic apply stage: findings in, ledger appends out —
|
||||||
|
* identity resolution, retroactive probe attribution, event rebuild, and
|
||||||
|
* the idempotency guarantees.
|
||||||
*
|
*
|
||||||
* Ledger isolation: PATHS is fixed at import time, so the five data/graph
|
* Ledger isolation: PATHS is fixed at import time, so the five data/graph
|
||||||
* ledgers are backed up and cleared before the suite and restored after
|
* ledgers are backed up and cleared before the suite and restored after
|
||||||
* (try/finally-shaped: the after() hook always runs, even on failure).
|
* (try/finally-shaped: the after() hook always runs, even on failure).
|
||||||
* The chat fixtures are real ChatRecords in the chat store (created and
|
* The chat fixtures are real ChatRecords in the chat store (created and
|
||||||
* deleted by this suite) — emission reads exactly what production reads.
|
* deleted by this suite) — the merger reads exactly what production reads.
|
||||||
|
* The full LLM pipeline (extractor → auditor) is tested in
|
||||||
|
* tests/extraction.test.ts with mock models.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import assert from "node:assert/strict";
|
import assert from "node:assert/strict";
|
||||||
import * as fs from "node:fs";
|
import * as fs from "node:fs";
|
||||||
import { after, describe, it } from "node:test";
|
import { after, describe, it } from "node:test";
|
||||||
import type { UIMessage } from "ai";
|
import type { UIMessage } from "ai";
|
||||||
import type { PlanTree, PlanToolResult } from "../src/core/contracts.ts";
|
import type { PlanTree, PlanToolResult, SessionFindings } from "../src/core/contracts.ts";
|
||||||
import type { QuestionAnswerData } from "../src/core/question-core.ts";
|
import type { QuestionAnswerData } from "../src/core/question-core.ts";
|
||||||
import { createTree, markDone, removeNode } from "../src/core/plan-tree.ts";
|
import { createTree, markDone } from "../src/core/plan-tree.ts";
|
||||||
import { emitChat, foldStaleChats, getGraph, graphSnapshot, type GraphDeps } from "../src/core/graph.ts";
|
import { getGraph, graphSnapshot, mergeFindings, pendingQuizAnswers, type GraphDeps } from "../src/core/graph.ts";
|
||||||
import { graphStore } from "../src/store/graph.ts";
|
import { graphStore } from "../src/store/graph.ts";
|
||||||
import { chatStore } from "../src/store/chats.ts";
|
import { chatStore } from "../src/store/chats.ts";
|
||||||
import { LEDGER_FILES, PATHS } from "../config.ts";
|
import { LEDGER_FILES, PATHS } from "../config.ts";
|
||||||
@@ -32,7 +38,7 @@ const createdChats: string[] = [];
|
|||||||
fs.mkdirSync(PATHS.graph, { recursive: true });
|
fs.mkdirSync(PATHS.graph, { recursive: true });
|
||||||
for (const file of LEDGER_PATHS) {
|
for (const file of LEDGER_PATHS) {
|
||||||
if (fs.existsSync(file)) {
|
if (fs.existsSync(file)) {
|
||||||
const bak = `${file}.w4test.bak`;
|
const bak = `${file}.w5test.bak`;
|
||||||
fs.renameSync(file, bak);
|
fs.renameSync(file, bak);
|
||||||
backups.push({ file, bak });
|
backups.push({ file, bak });
|
||||||
}
|
}
|
||||||
@@ -120,6 +126,32 @@ function fixtureTree(): PlanTree {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Findings shaped exactly like what the extractor would produce for a
|
||||||
|
* session's plan tree: one proposition per tree node (statement = title,
|
||||||
|
* treeNodeId set), one `requires` edge per tree edge.
|
||||||
|
*/
|
||||||
|
function treeFindings(tree: PlanTree): SessionFindings {
|
||||||
|
const idByNodeId = new Map(tree.nodes.map((n, i) => [n.id, `p${i + 1}`]));
|
||||||
|
return {
|
||||||
|
propositions: tree.nodes.map((n, i) => ({
|
||||||
|
id: `p${i + 1}`,
|
||||||
|
statement: n.title,
|
||||||
|
aliases: [],
|
||||||
|
kind: "truth" as const,
|
||||||
|
treeNodeId: n.id,
|
||||||
|
evidence: ["fixture evidence"],
|
||||||
|
})),
|
||||||
|
edges: tree.edges.map((e) => ({
|
||||||
|
from: idByNodeId.get(e.from)!,
|
||||||
|
to: idByNodeId.get(e.to)!,
|
||||||
|
type: "requires" as const,
|
||||||
|
evidence: ["fixture evidence"],
|
||||||
|
})),
|
||||||
|
attribution: { quiz: {} },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
/** The full fixture transcript: probes → plan → checks → mark_done → noise. */
|
/** The full fixture transcript: probes → plan → checks → mark_done → noise. */
|
||||||
function fixtureMessages(tree: PlanTree): UIMessage[] {
|
function fixtureMessages(tree: PlanTree): UIMessage[] {
|
||||||
return [
|
return [
|
||||||
@@ -154,7 +186,8 @@ function fixtureMessages(tree: PlanTree): UIMessage[] {
|
|||||||
),
|
),
|
||||||
]),
|
]),
|
||||||
assistant("a6", [
|
assistant("a6", [
|
||||||
// nodeId present but NOT in the tree → unmapped → null + question text
|
// nodeId present but NOT in the tree → tagged-but-unmapped: the
|
||||||
|
// attribution can rescue it, or it lands null + question text.
|
||||||
quizPart(
|
quizPart(
|
||||||
"call-5",
|
"call-5",
|
||||||
{ question: "Q5", nodeId: "ghost", options: [{ label: "a" }, { label: "b" }], correctAnswer: "a", explanation: "e" },
|
{ question: "Q5", nodeId: "ghost", options: [{ label: "a" }, { label: "b" }], correctAnswer: "a", explanation: "e" },
|
||||||
@@ -163,7 +196,7 @@ function fixtureMessages(tree: PlanTree): UIMessage[] {
|
|||||||
]),
|
]),
|
||||||
assistant("a7", [planPart("p2", "plan_mark_done", { id: "alpha" }, tree)]),
|
assistant("a7", [planPart("p2", "plan_mark_done", { id: "alpha" }, tree)]),
|
||||||
assistant("a8", [
|
assistant("a8", [
|
||||||
// dismissed quiz + an answered ask — neither emits an event
|
// dismissed quiz + an answered ask — neither becomes an event
|
||||||
quizPart(
|
quizPart(
|
||||||
"call-6",
|
"call-6",
|
||||||
{ question: "Q6", options: [{ label: "a" }, { label: "b" }], correctAnswer: "a", explanation: "e" },
|
{ question: "Q6", options: [{ label: "a" }, { label: "b" }], correctAnswer: "a", explanation: "e" },
|
||||||
@@ -180,13 +213,27 @@ function fixtureMessages(tree: PlanTree): UIMessage[] {
|
|||||||
];
|
];
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The chat every emission test folds (shared, so ordering is intentional). */
|
/**
|
||||||
|
* The extractor's findings for the fixture chat: the tree as propositions +
|
||||||
|
* the probe attribution. The audit stage is a no-op pass-through in these
|
||||||
|
* merger tests — its behavior is tested with the pipeline (extraction.test).
|
||||||
|
*/
|
||||||
|
function fixtureFindings(): SessionFindings {
|
||||||
|
const findings = treeFindings(fixtureTree());
|
||||||
|
// Untagged probe quizzes: call-1 attributed retroactively to Alpha (the
|
||||||
|
// vision's open question — resolved by the extractor); call-2 genuinely
|
||||||
|
// unattributable; call-5's ghost nodeId is rescued onto Gamma.
|
||||||
|
findings.attribution.quiz = { "call-1": "p1", "call-2": null, "call-5": "p3" };
|
||||||
|
return findings;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The chat every merger test folds (shared, so ordering is intentional). */
|
||||||
const chat1 = fixtureChat("Graph fixture", fixtureTree(), fixtureMessages(fixtureTree()));
|
const chat1 = fixtureChat("Graph fixture", fixtureTree(), fixtureMessages(fixtureTree()));
|
||||||
|
|
||||||
/** Set by the provenance test; the snapshot test (which runs after it) reads it. */
|
/** Set by the cross-session test; the snapshot test (which runs after it) reads it. */
|
||||||
let chat2Id = "";
|
let chat2Id = "";
|
||||||
|
|
||||||
// ─── emission ────────────────────────────────────────────────────────────────
|
// ─── the merger ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
/** Everything the graph holds for a set of chats (other processes' test
|
/** Everything the graph holds for a set of chats (other processes' test
|
||||||
* emissions run concurrently against the same files — scope, never assume). */
|
* emissions run concurrently against the same files — scope, never assume). */
|
||||||
@@ -205,9 +252,9 @@ function scopedGraph(chatIds: string[]) {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
describe("graph emission (emitChat)", () => {
|
describe("the merger (mergeFindings)", () => {
|
||||||
it("folds tree nodes/edges and quiz/mark_done events with the probe/check boundary", () => {
|
it("merges propositions/edges by identity and rebuilds events with retroactive probe attribution", () => {
|
||||||
const stats = emitChat(chat1.id, graphDeps);
|
const stats = mergeFindings(chat1.id, fixtureFindings(), graphDeps);
|
||||||
assert.deepEqual(stats, { nodes: 3, edges: 1, events: 6 });
|
assert.deepEqual(stats, { nodes: 3, edges: 1, events: 6 });
|
||||||
|
|
||||||
const graph = scopedGraph([chat1.id]);
|
const graph = scopedGraph([chat1.id]);
|
||||||
@@ -219,7 +266,7 @@ describe("graph emission (emitChat)", () => {
|
|||||||
assert.match(node.id, /^n_/);
|
assert.match(node.id, /^n_/);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Tree edge alpha→beta becomes a `requires` edge on the mapped nodes.
|
// The tree edge alpha→beta becomes a `requires` edge on the mapped nodes.
|
||||||
assert.equal(graph.edges.length, 1);
|
assert.equal(graph.edges.length, 1);
|
||||||
const edge = graph.edges[0]!;
|
const edge = graph.edges[0]!;
|
||||||
assert.equal(edge.from, idByStatement.get("Alpha intuition"));
|
assert.equal(edge.from, idByStatement.get("Alpha intuition"));
|
||||||
@@ -227,77 +274,69 @@ describe("graph emission (emitChat)", () => {
|
|||||||
assert.equal(edge.type, "requires");
|
assert.equal(edge.type, "requires");
|
||||||
assert.deepEqual(edge.provenance, [chat1.id]);
|
assert.deepEqual(edge.provenance, [chat1.id]);
|
||||||
|
|
||||||
// Events — kinds, refs, node attribution, notes, question fallback.
|
// Events — kinds, refs, notes, and the attribution paths.
|
||||||
assert.equal(graph.events.length, 6);
|
assert.equal(graph.events.length, 6);
|
||||||
const byRef = new Map(graph.events.map((e) => [e.ref, e]));
|
const byRef = new Map(graph.events.map((e) => [e.ref, e]));
|
||||||
|
|
||||||
const probeMiss = byRef.get("call-1")!;
|
const probeMiss = byRef.get("call-1")!;
|
||||||
assert.equal(probeMiss.kind, "probe-miss"); // quiz BEFORE plan_init
|
assert.equal(probeMiss.kind, "probe-miss"); // quiz BEFORE plan_init
|
||||||
assert.equal(probeMiss.nodeId, null);
|
assert.equal(probeMiss.nodeId, idByStatement.get("Alpha intuition"), "retroactive probe attribution");
|
||||||
assert.equal(probeMiss.question, "Q1 text");
|
assert.equal(probeMiss.question, undefined, "attributed → no question text needed");
|
||||||
assert.equal(probeMiss.chatId, chat1.id);
|
|
||||||
|
|
||||||
const probeDontKnow = byRef.get("call-2")!;
|
const probeDontKnow = byRef.get("call-2")!;
|
||||||
assert.equal(probeDontKnow.kind, "dont-know"); // same kind, both phases
|
assert.equal(probeDontKnow.kind, "dont-know"); // same kind, both phases
|
||||||
assert.equal(probeDontKnow.nodeId, null);
|
assert.equal(probeDontKnow.nodeId, null, "unattributable probe → null");
|
||||||
|
assert.equal(probeDontKnow.question, "Q2 text");
|
||||||
|
|
||||||
const check = byRef.get("call-3")!;
|
const check = byRef.get("call-3")!;
|
||||||
assert.equal(check.kind, "check-correct"); // quiz AFTER plan_init
|
assert.equal(check.kind, "check-correct"); // quiz AFTER plan_init
|
||||||
assert.equal(check.nodeId, idByStatement.get("Alpha intuition"));
|
assert.equal(check.nodeId, idByStatement.get("Alpha intuition"), "tagged quiz → deterministic");
|
||||||
assert.equal(check.note, "recalled the diagram");
|
assert.equal(check.note, "recalled the diagram");
|
||||||
assert.equal(check.question, undefined); // resolved nodeId → no question text
|
|
||||||
|
|
||||||
const checkMiss = byRef.get("call-4")!;
|
const checkMiss = byRef.get("call-4")!;
|
||||||
assert.equal(checkMiss.kind, "check-incorrect");
|
assert.equal(checkMiss.kind, "check-incorrect");
|
||||||
assert.equal(checkMiss.nodeId, idByStatement.get("Beta rule"));
|
assert.equal(checkMiss.nodeId, idByStatement.get("Beta rule"));
|
||||||
|
|
||||||
const unmapped = byRef.get("call-5")!;
|
const ghost = byRef.get("call-5")!;
|
||||||
assert.equal(unmapped.kind, "dont-know");
|
assert.equal(ghost.kind, "dont-know");
|
||||||
assert.equal(unmapped.nodeId, null); // nodeId present but unmapped
|
assert.equal(ghost.nodeId, idByStatement.get("Gamma application"), "tagged-but-unmapped rescued by attribution");
|
||||||
assert.equal(unmapped.question, "Q5 text");
|
assert.equal(ghost.question, undefined);
|
||||||
|
|
||||||
const taught = byRef.get("taught:alpha")!;
|
const taught = byRef.get("taught:alpha")!;
|
||||||
assert.equal(taught.kind, "taught");
|
assert.equal(taught.kind, "taught");
|
||||||
assert.equal(taught.nodeId, idByStatement.get("Alpha intuition"));
|
assert.equal(taught.nodeId, idByStatement.get("Alpha intuition"), "mark_done → tree-node mapping");
|
||||||
|
|
||||||
// The dismissed quiz and the ask question emitted nothing.
|
// The dismissed quiz and the ask question became nothing.
|
||||||
assert.equal([...byRef.keys()].filter((r) => r === "call-6" || r === "call-7").length, 0);
|
assert.equal([...byRef.keys()].filter((r) => r === "call-6" || r === "call-7").length, 0);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("is idempotent — re-emission appends nothing", () => {
|
it("is idempotent — re-merging the same findings appends nothing", () => {
|
||||||
const stats = emitChat(chat1.id, graphDeps);
|
const stats = mergeFindings(chat1.id, fixtureFindings(), graphDeps);
|
||||||
assert.deepEqual(stats, { nodes: 0, edges: 0, events: 0 });
|
assert.deepEqual(stats, { nodes: 0, edges: 0, events: 0 });
|
||||||
const graph = scopedGraph([chat1.id]);
|
const graph = scopedGraph([chat1.id]);
|
||||||
assert.equal(graph.nodes.length, 3);
|
assert.equal(graph.nodes.length, 3);
|
||||||
assert.equal(graph.edges.length, 1);
|
assert.equal(graph.edges.length, 1);
|
||||||
assert.equal(graph.events.length, 6);
|
assert.equal(graph.events.length, 6);
|
||||||
// exactly one checkpoint line per fold, though — checkpoints are a log
|
// exactly one checkpoint line per merge, though — checkpoints are a log
|
||||||
assert.equal(graphStore.readCheckpoints().filter((c) => c.chatId === chat1.id).length, 2);
|
assert.equal(graphStore.readCheckpoints().filter((c) => c.chatId === chat1.id).length, 2);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("accumulates edge provenance when another chat maps onto the same nodes", () => {
|
it("merges a second session onto the same nodes and accumulates edge provenance", () => {
|
||||||
// Simulates what a future consolidation merge produces: chat2's tree
|
// chat2 re-teaches the same two propositions (identical statements) —
|
||||||
// slugs map onto chat1's graph nodes, so its edge assertion dedupes
|
// the conservative identity resolution recognizes them.
|
||||||
// by (from,to,type) and just adds chat2 to the provenance.
|
|
||||||
const tree2 = createTree(
|
const tree2 = createTree(
|
||||||
[
|
[
|
||||||
{ id: "alpha", title: "Alpha (second session)" },
|
{ id: "a", title: "Alpha intuition" },
|
||||||
{ id: "beta", title: "Beta (second session)" },
|
{ id: "b", title: "Beta rule" },
|
||||||
],
|
],
|
||||||
[{ from: "alpha", to: "beta" }],
|
[{ from: "a", to: "b" }],
|
||||||
);
|
);
|
||||||
const chat2 = fixtureChat("Graph fixture 2", tree2, [
|
const chat2 = fixtureChat("Graph fixture 2", tree2, [
|
||||||
user("u1", "again"),
|
user("u1", "again"),
|
||||||
assistant("a1", [planPart("p1", "plan_init", { nodes: [], edges: [] }, tree2)]),
|
assistant("a1", [planPart("p1", "plan_init", { nodes: [], edges: [] }, tree2)]),
|
||||||
]);
|
]);
|
||||||
|
|
||||||
const before = getGraph(graphStore);
|
const stats = mergeFindings(chat2.id, treeFindings(tree2), graphDeps);
|
||||||
const gAlpha = before.nodes.find((n) => n.statement === "Alpha intuition")!.id;
|
|
||||||
const gBeta = before.nodes.find((n) => n.statement === "Beta rule")!.id;
|
|
||||||
graphStore.appendMapping({ chatId: chat2.id, treeNodeId: "alpha", graphNodeId: gAlpha });
|
|
||||||
graphStore.appendMapping({ chatId: chat2.id, treeNodeId: "beta", graphNodeId: gBeta });
|
|
||||||
|
|
||||||
const stats = emitChat(chat2.id, graphDeps);
|
|
||||||
assert.deepEqual(stats, { nodes: 0, edges: 1, events: 0 });
|
assert.deepEqual(stats, { nodes: 0, edges: 1, events: 0 });
|
||||||
chat2Id = chat2.id;
|
chat2Id = chat2.id;
|
||||||
|
|
||||||
@@ -312,7 +351,9 @@ describe("graph emission (emitChat)", () => {
|
|||||||
const idByStatement = new Map(snap.nodes.map((n) => [n.statement, n.id]));
|
const idByStatement = new Map(snap.nodes.map((n) => [n.statement, n.id]));
|
||||||
assert.equal(snap.nodeStates[idByStatement.get("Alpha intuition")!], "taught");
|
assert.equal(snap.nodeStates[idByStatement.get("Alpha intuition")!], "taught");
|
||||||
assert.equal(snap.nodeStates[idByStatement.get("Beta rule")!], "missed");
|
assert.equal(snap.nodeStates[idByStatement.get("Beta rule")!], "missed");
|
||||||
assert.equal(snap.nodeStates[idByStatement.get("Gamma application")!], "candidate");
|
// Gamma: no taught/checks — but the rescued dont-know probe (call-5)
|
||||||
|
// makes its latest event a dont-know.
|
||||||
|
assert.equal(snap.nodeStates[idByStatement.get("Gamma application")!], "dont-know");
|
||||||
|
|
||||||
assert.equal(snap.sessions[chat1.id], "Graph fixture");
|
assert.equal(snap.sessions[chat1.id], "Graph fixture");
|
||||||
assert.equal(snap.sessions[chat2Id], "Graph fixture 2");
|
assert.equal(snap.sessions[chat2Id], "Graph fixture 2");
|
||||||
@@ -320,44 +361,25 @@ describe("graph emission (emitChat)", () => {
|
|||||||
(snap.nodeSessions[idByStatement.get("Alpha intuition")!] ?? []).sort(),
|
(snap.nodeSessions[idByStatement.get("Alpha intuition")!] ?? []).sort(),
|
||||||
[chat1.id, chat2Id].sort(),
|
[chat1.id, chat2Id].sort(),
|
||||||
);
|
);
|
||||||
// Gamma was only ever created (mapping) by chat1.
|
// Gamma was only ever touched by chat1.
|
||||||
assert.deepEqual(snap.nodeSessions[idByStatement.get("Gamma application")!], [chat1.id]);
|
assert.deepEqual(snap.nodeSessions[idByStatement.get("Gamma application")!], [chat1.id]);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("replays plan tool outputs when record.tree is missing (removed nodes stay out)", () => {
|
|
||||||
const initTree = createTree(
|
|
||||||
[
|
|
||||||
{ id: "x", title: "X live" },
|
|
||||||
{ id: "y", title: "Y pruned" },
|
|
||||||
],
|
|
||||||
[{ from: "x", to: "y" }],
|
|
||||||
);
|
|
||||||
const prunedTree = removeNode(initTree, "y");
|
|
||||||
const chat = fixtureChat("Replay fixture", null, [
|
|
||||||
user("u1", "go"),
|
|
||||||
assistant("a1", [planPart("p1", "plan_init", { nodes: [], edges: [] }, initTree)]),
|
|
||||||
assistant("a2", [planPart("p2", "plan_remove_node", { id: "y" }, prunedTree)]),
|
|
||||||
]);
|
|
||||||
|
|
||||||
const stats = emitChat(chat.id, graphDeps);
|
|
||||||
assert.deepEqual(stats, { nodes: 1, edges: 0, events: 0 }); // y pruned, edge cascaded away
|
|
||||||
|
|
||||||
const graph = scopedGraph([chat.id]);
|
|
||||||
assert.ok(graph.nodes.some((n) => n.statement === "X live"));
|
|
||||||
assert.ok(!graph.nodes.some((n) => n.statement === "Y pruned"));
|
|
||||||
});
|
|
||||||
|
|
||||||
it("returns zeros for unknown chats", () => {
|
it("returns zeros for unknown chats", () => {
|
||||||
assert.deepEqual(emitChat("chat_does_not_exist", graphDeps), { nodes: 0, edges: 0, events: 0 });
|
assert.deepEqual(mergeFindings("chat_does_not_exist", treeFindings(fixtureTree()), graphDeps), {
|
||||||
|
nodes: 0,
|
||||||
|
edges: 0,
|
||||||
|
events: 0,
|
||||||
|
});
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// chat2 from the provenance test, reachable by the snapshot test below it.
|
// ─── the trigger helper ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
describe("catch-up (foldStaleChats)", () => {
|
describe("the trigger helper (pendingQuizAnswers)", () => {
|
||||||
it("folds chats newer than their checkpoint, exactly once, deriving known state", () => {
|
it("counts answered quizzes not yet in the ledger — dismissed/ask never count", () => {
|
||||||
const tree = createTree([{ id: "solo", title: "Solo node" }], []);
|
const tree = createTree([{ id: "solo", title: "Solo node" }], []);
|
||||||
const chat = fixtureChat("Catch-up fixture", tree, [
|
const chat = fixtureChat("Trigger fixture", tree, [
|
||||||
user("u1", "go"),
|
user("u1", "go"),
|
||||||
assistant("a1", [planPart("p1", "plan_init", { nodes: [], edges: [] }, tree)]),
|
assistant("a1", [planPart("p1", "plan_init", { nodes: [], edges: [] }, tree)]),
|
||||||
assistant("a2", [
|
assistant("a2", [
|
||||||
@@ -366,23 +388,39 @@ describe("catch-up (foldStaleChats)", () => {
|
|||||||
{ question: "QS", nodeId: "solo", options: [{ label: "a" }, { label: "b" }], correctAnswer: "a", explanation: "e" },
|
{ question: "QS", nodeId: "solo", options: [{ label: "a" }, { label: "b" }], correctAnswer: "a", explanation: "e" },
|
||||||
answer({ questionId: "call-s", questionText: "QS text", correct: true, answeredAt: "2026-08-22T03:00:00.000Z" }),
|
answer({ questionId: "call-s", questionText: "QS text", correct: true, answeredAt: "2026-08-22T03:00:00.000Z" }),
|
||||||
),
|
),
|
||||||
|
quizPart(
|
||||||
|
"call-s2",
|
||||||
|
{ question: "QS2", options: [{ label: "a" }, { label: "b" }], correctAnswer: "a", explanation: "e" },
|
||||||
|
answer({ questionId: "call-s2", questionText: "QS2 text", dismissed: true }),
|
||||||
|
),
|
||||||
|
{
|
||||||
|
type: "tool-ask_user_question",
|
||||||
|
toolCallId: "call-s3",
|
||||||
|
state: "output-available",
|
||||||
|
input: { question: "free?" },
|
||||||
|
output: answer({ questionId: "call-s3", kind: "ask", questionText: "free?", text: "y" }),
|
||||||
|
},
|
||||||
]),
|
]),
|
||||||
]);
|
]);
|
||||||
|
|
||||||
const folded = foldStaleChats(graphDeps);
|
assert.equal(pendingQuizAnswers(chat.id, graphDeps), 1, "one answered quiz pending");
|
||||||
assert.ok(folded >= 1, "the catch-up fixture was folded");
|
assert.equal(pendingQuizAnswers("chat_does_not_exist", graphDeps), 0);
|
||||||
assert.equal(graphStore.readCheckpoints().filter((c) => c.chatId === chat.id).length, 1);
|
|
||||||
|
|
||||||
const snap = graphSnapshot(graphDeps);
|
mergeFindings(chat.id, treeFindings(tree), graphDeps);
|
||||||
const solo = snap.nodes.find((n) => n.statement === "Solo node");
|
assert.equal(pendingQuizAnswers(chat.id, graphDeps), 0, "extracted → nothing pending");
|
||||||
assert.ok(solo, "solo node was emitted by the catch-up fold");
|
|
||||||
assert.equal(snap.nodeStates[solo.id], "known"); // correct check, no taught event
|
|
||||||
assert.equal(snap.sessions[chat.id], "Catch-up fixture");
|
|
||||||
|
|
||||||
// Already checkpointed → not folded again (other test files' chats may
|
// A NEW answered quiz raises the count again (the re-trigger path).
|
||||||
// legitimately fold here, so assert on THIS chat only).
|
chatStore.saveMessages(chat.id, [
|
||||||
foldStaleChats(graphDeps);
|
...chatStore.getChat(chat.id)!.messages,
|
||||||
assert.equal(graphStore.readCheckpoints().filter((c) => c.chatId === chat.id).length, 1);
|
assistant("a3", [
|
||||||
|
quizPart(
|
||||||
|
"call-s4",
|
||||||
|
{ question: "QS4", nodeId: "solo", options: [{ label: "a" }, { label: "b" }], correctAnswer: "a", explanation: "e" },
|
||||||
|
answer({ questionId: "call-s4", questionText: "QS4 text", correct: false, answeredAt: "2026-08-22T03:01:00.000Z" }),
|
||||||
|
),
|
||||||
|
]),
|
||||||
|
]);
|
||||||
|
assert.equal(pendingQuizAnswers(chat.id, graphDeps), 1);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user