From e944afebfce07b830cb804e3617acb6733cf8b09 Mon Sep 17 00:00:00 2001 From: amosblomqvist Date: Wed, 9 Sep 2026 02:06:32 +0300 Subject: [PATCH] Understanding levels: closed five-word scale, schema-enforced MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The extractor's per-session understanding notes were free-form strings; the model freely appended em-dash justifications ('shaky — the learner could state the headline distinction but calls the machinery behind it foggy'), which overflowed the detail card's session rows and squeezed the session names. The note becomes a CLOSED named-word scale — solid / shaky / accepted / resisted / not yet engaged (DECIDED 2026-09-08, no numbers) — enforced by the zod enum in propositionSchema: a wrong-shaped note is rejected at parse time and the agent retries. Justifications are dropped, not stored (user decision: not necessary). Extractor and auditor instructions updated to demand exactly one word. Renderer: the session row keeps the session NAME as the main thing (flex-1, truncates) and the level chip is bounded (max-w-28, whitespace-nowrap, truncate) — overflow impossible even for legacy prose notes in old ledgers. Vision doc: §1 extractor bullet + event-ledger line updated; state model line notes the five-class derivation was removed; amendments table row added. --- docs/vision.md | 7 ++++--- src/agent/subagents/extraction.ts | 17 ++++++++++------- src/core/contracts/graph.ts | 23 +++++++++++++++++------ tests/graph.test.ts | 8 ++++---- 4 files changed, 35 insertions(+), 20 deletions(-) diff --git a/docs/vision.md b/docs/vision.md index d722f9f..a220a41 100644 --- a/docs/vision.md +++ b/docs/vision.md @@ -43,15 +43,15 @@ Every node records every **(chat, message) point** where it came up; every edge Three stages, one pipeline — the sole graph writer: -- **Extractor (LLM) — session-only scope.** Sees ONLY the session transcript (tree-blind, with message ids). Reports freely, with verbatim evidence: propositions from anywhere in the session (one reasoning step each; prose counts as much as anything the teacher formalized), edges as the transcript articulates them, message-granular provenance, retroactive quiz attribution, and per proposition an **understanding note** — how strongly the learner accepted/understood it in this session, a small named-word scale, no numbers. The notes are data for the future state model; their treatment is TBD. +- **Extractor (LLM) — session-only scope.** Sees ONLY the session transcript (tree-blind, with message ids). Reports freely, with verbatim evidence: propositions from anywhere in the session (one reasoning step each; prose counts as much as anything the teacher formalized), edges as the transcript articulates them, message-granular provenance, retroactive quiz attribution, and per proposition an **understanding level** — how strongly the learner accepted/understood it in this session, a CLOSED named-word scale (`solid` / `shaky` / `accepted` / `resisted` / `not yet engaged`; 2026-09-08: schema-enforced — exactly one word, no numbers, no justifications, which are dropped rather than stored). The levels are data for the future state model; their treatment is TBD. - **Auditor (LLM) — the global-graph scope.** Sees the extractor's output AND the existing graph. Two jobs: the faithfulness net (every candidate verified against verbatim evidence — unchanged), and the **identity judgment**: link each proposition to an existing node (recording the session's phrasing as an alias) or create it new. Conservative: duplicates over wrong merges. - **Merger (deterministic).** Append-only ledger writes; quiz event kinds/refs rebuilt mechanically from quiz tool parts. Every judgment is an LLM's; everything mechanical stays code. -**Event ledger (amended):** quiz outcomes are rebuilt deterministically from quiz tool parts (correct / miss / don't-know, plus the learner's note). **There is no `taught` event** — no marking-things-done. The extractor's per-proposition understanding notes join the ledger as session-scoped observations. Identity is no longer exact-string matching — it is the auditor's judgment, with aliases as the phrasing record. +**Event ledger (amended):** quiz outcomes are rebuilt deterministically from quiz tool parts (correct / miss / don't-know, plus the learner's note). **There is no `taught` event** — no marking-things-done. The extractor's per-proposition understanding levels join the ledger as session-scoped observations (one closed-scale word per node per session). Identity is no longer exact-string matching — it is the auditor's judgment, with aliases as the phrasing record. ### Understanding is derived (amended 2026-09-02) -Known vs. understood, event ledger as source of truth, derived state recomputed on read — carried over from the design doc in spirit. The state model itself is now an explicit design discussion: **node states should be session-independent and computed from the depth and strength of dependencies** (the `follows`/`supports` topology) plus quiz events and understanding notes. That redesign is pending ideation; the current five-class derivation (taught / known / missed / don't-know / candidate) is transitional. The anchored calibration rubric stays parked until real data. +Known vs. understood, event ledger as source of truth, derived state recomputed on read — carried over from the design doc in spirit. The state model itself is now an explicit design discussion: **node states should be session-independent and computed from the depth and strength of dependencies** (the `follows`/`supports` topology) plus quiz events and understanding notes. That redesign is pending ideation; the transitional five-class derivation (taught / known / missed / don't-know / candidate) was **removed** (2026-09-08) — the graph view renders state-neutral nodes until the state model is designed. The anchored calibration rubric stays parked until real data. ### Rendering (DECIDED — research report 2026-08-22) @@ -224,6 +224,7 @@ The teacher's context window and attention are precious — it has the lesson to | Misconception nodes as first-class structure | No node `kind`: every node is a true proposition; misconceptions enter as true corrections, wrongness lives in the event ledger | | `requires` edges | `follows` / `supports` — derivability vs. grounding; strength derived at read time, never stored | | `taught` events (marking things done) | Dropped; the extractor records per-proposition understanding notes (named-word scale) as session-scoped observations instead | +| Understanding scale standardized (2026-09-08) | The note becomes a closed five-word scale (`solid`/`shaky`/`accepted`/`resisted`/`not yet engaged`), enforced by the extraction schema — no numbers, no prose; agent justifications are dropped | | Chat-granular node/edge provenance | Message-granular (chat, message) provenance on nodes and edges — the exact derivation points | | `name` + `statement` on nodes | `statement` (minimal proposition, the rendering label) + `description` (required elaboration) + `aliases` (verbatim phrasings) | | `statement` + `description` (2026-09-02) | `name` (very short label, the rendering handle) + `statement` (full precision, identity anchor, detail card); `description` dropped — full-sentence canvas labels cluttered the graph (2026-09-08, seeing real extracted data) | diff --git a/src/agent/subagents/extraction.ts b/src/agent/subagents/extraction.ts index 105e9e1..629d221 100644 --- a/src/agent/subagents/extraction.ts +++ b/src/agent/subagents/extraction.ts @@ -118,11 +118,12 @@ const propositionSchema = z.object({ messages: messagesField, evidence, understanding: z - .string() + .enum(["solid", "shaky", "accepted", "resisted", "not yet engaged"]) .optional() .describe( - 'One note about THIS session\'s evidence of the learner\'s grasp of this proposition, on a named-word scale ' + - '("solid", "shaky", "accepted", "resisted", "not yet engaged") — no numbers, no predictions.', + "ONE word — the learner's grasp of this proposition in THIS session, from the fixed scale " + + '"solid" | "shaky" | "accepted" | "resisted" | "not yet engaged". Nothing else: no numbers, ' + + "no justification, no explanation.", ), }); @@ -180,8 +181,9 @@ quiz outcomes carry that the learner believed the wrong version. - One reasoning step per proposition: never a topic blob, never a sub-step. - "messages": the [id] tags of every message where the proposition came up — provenance, report it. - "aliases": verbatim phrasings this session used for the same proposition. - - "understanding": ONE named-word-scale note on the evidence of the learner's grasp, for this session - only ("solid", "shaky", "accepted", "resisted", "not yet engaged"). No numbers, no predictions. + - "understanding": ONE word from the fixed scale — "solid", "shaky", "accepted", "resisted", or + "not yet engaged" — judging the learner's grasp of the proposition in THIS session. Strictly one + word, no numbers, no justification, no explanation (a wrong-shaped note is rejected by the schema). 2. EDGES between propositions: - "follows": to is DERIVABLE from from (understanding from makes understanding to achievable). @@ -256,8 +258,9 @@ FAITHFULNESS — verify every candidate against the transcript: direction-swapped or speculative edges. - ATTRIBUTION: the QUIZ line's question must genuinely test the attributed proposition. Drop wrong attributions to null, or re-point them to the proposition the question actually tests. -- UNDERSTANDING notes: verify against the quiz outcomes and the session's evidence; reword to a named-word - scale or drop. +- UNDERSTANDING levels: verify against the quiz outcomes and the session's evidence; each note must be + exactly one word from the fixed scale ("solid", "shaky", "accepted", "resisted", "not yet engaged") — + reword or drop anything else. No justifications. IDENTITY — link-or-create, per proposition: - You see the current graph (every node: id, name, statement, aliases). For each verified proposition, diff --git a/src/core/contracts/graph.ts b/src/core/contracts/graph.ts index 158b320..a0ba30d 100644 --- a/src/core/contracts/graph.ts +++ b/src/core/contracts/graph.ts @@ -102,7 +102,8 @@ export interface LedgerEvent { * understanding observations — re-emission of the same chat must never * double-record an event. */ ref?: string; - /** The learner's quiz note — diagnostic gold (vision §1 calibration). */ + /** The learner's quiz note (diagnostic gold), or the understanding + * LEVEL word for observations — a session-relative judgment. */ note?: string; /** The quiz question text, when nodeId is null (later attribution input). */ question?: string; @@ -138,6 +139,16 @@ export interface LedgerEvent { // ─── Extraction findings (extractor output → audit → merger input) ─────────── +/** + * The extractor's standardized per-session understanding scale (DECIDED + * 2026-09-08): a CLOSED named-word list — no numbers, no prose, no + * justifications (a justification, if the agent wants one, is dropped; + * it is not stored). The level is session- and extraction-run-relative + * judgment about the learner, shown next to its session — never a node + * property. + */ +export type UnderstandingLevel = "solid" | "shaky" | "accepted" | "resisted" | "not yet engaged"; + /** One proposed proposition — a node candidate (or a re-encounter of one). */ export interface ExtractedProposition { /** Extractor-local id ("p1", …) referenced by edges + attribution. */ @@ -158,11 +169,11 @@ export interface ExtractedProposition { messages: string[]; /** Verbatim transcript quotes backing the proposition. */ evidence: string[]; - /** The extractor's understanding note for this proposition IN THIS - * session — a named-word scale ("solid", "shaky", "accepted", - * "resisted"), no numbers. Joins the ledger as an observation event; - * treatment pending the state-model ideation. */ - understanding?: string; + /** The extractor's understanding level for this proposition IN THIS + * session — one word from the UnderstandingLevel scale. Joins the + * ledger as an observation event (note = the level word); treatment + * pending the state-model ideation. */ + understanding?: UnderstandingLevel; } /** One proposed relation between two propositions. */ diff --git a/tests/graph.test.ts b/tests/graph.test.ts index f440ab6..2a104d1 100644 --- a/tests/graph.test.ts +++ b/tests/graph.test.ts @@ -172,7 +172,7 @@ function fixtureAudit(): AuditResult { aliases: ["alpha feel"], messages: ["a1", "a3"], evidence: ["fixture evidence"], - understanding: "solid after the check", + understanding: "solid", }, { id: "p2", @@ -295,7 +295,7 @@ describe("the merger (mergeFindings)", () => { const observation = byRef.get("obs:p1")!; assert.equal(observation.kind, "observation"); assert.equal(observation.nodeId, idByStatement.get("Alpha intuition")); - assert.equal(observation.note, "solid after the check"); + assert.equal(observation.note, "solid"); // The dismissed quiz and the ask question became nothing. assert.equal([...byRef.keys()].filter((r) => r === "call-6" || r === "call-7").length, 0); @@ -380,9 +380,9 @@ describe("the merger (mergeFindings)", () => { assert.deepEqual(snap.nodeSessions[idByStatement.get("Gamma application")!], [chat1.id]); // Understanding observations: (node → chat → the extractor's note). - // Alpha carries chat1's "solid after the check"; chat2's q1 has no note. + // Alpha carries chat1's "solid" level; chat2's q1 has no level. const alphaObs = snap.nodeObservations[idByStatement.get("Alpha intuition")!] ?? {}; - assert.equal(alphaObs[chat1.id], "solid after the check"); + assert.equal(alphaObs[chat1.id], "solid"); assert.equal(alphaObs[chat2Id], undefined); });