From 25cc2a238a8882335e667a8215d066f7bf8802be Mon Sep 17 00:00:00 2001 From: amosblomqvist Date: Tue, 29 Sep 2026 20:21:12 +0300 Subject: [PATCH] =?UTF-8?q?practice=20=E2=80=94=20formatting=20doctrine=20?= =?UTF-8?q?for=20generator=20+=20grader=20(owner-approved=202026-09-29)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prior state: zero formatting guidance anywhere in the agent module — models wrote prose-only by default. GENERATOR_DOCTRINE Part III gains the FORMATTING bullet (after the THE PARAMETERS bullet, before PREVIOUS PROBLEMS): statement, solution, and rubric render as full Markdown — math always LaTeX (inline $…$, display $$…$$), code/pseudocode/formal systems in fenced blocks with a language tag, identifiers inline as `code`, tables for parallel structure, lists for enumerations; prose-only renderings of mathematical or structured content are a defect. GRADER_INSTRUCTIONS gains the feedback-formatting paragraph (before the OUTPUT section): full Markdown, LaTeX math, fenced code, lists/ tables where they carry structure — teach the boundary in properly formatted prose, not plain text. PARTITIONER gets nothing (machine-shaped output). Tests: substring pins on the exported doctrine constants — the exact FORMATTING sentence + ordering (after presentation, before previous problems), the exact grader paragraph + before-OUTPUT ordering. Architecture.md §7 one clause (the formatting doctrine in the generator's prompt inventory). 1144/1144, tsc 0, lint 0, build 0. --- docs/architecture.md | 2 +- src/agent/subagents/practice.ts | 7 ++++-- tests/practice-agents.test.ts | 41 +++++++++++++++++++++++++++++++++ 3 files changed, 47 insertions(+), 3 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 9bee9c0..0b81c50 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -376,7 +376,7 @@ Practice turns the graph into problem sets the learner reasons through out loud. **The three axes are discretized named levels** (owner, 2026-09-29 — never a number to interpolate; there is no continuum between levels): the **ladder rung** (the main axis — 0 Worked example, 1 Completion, 2 Direct exercise, 3 Composition, 4 Path-finding; the invocation names the rung and the generator builds exactly that CLASS of problem), the **cue distance** (0 Lesson context — the textbook/toy-example shape in the teaching's own framing; 1 Adjacent context — a genuinely new situation built from the teaching's own world; 2 Far context — the edge of the recognizable band, where the proposition must still genuinely apply and be findable by looking), and the **presentation** (Theory — the subject's own terms, or Application — a concrete worldly affair in which the proposition's structure appears in the wild; it dresses the situation, never changes the rung or the cue distance). The names + one-line definitions live in one place (`src/core/practice.ts`'s level tables) and are spoken by everything: the agents' prompts (`renderParams`), the creation dialog's stepped controls, the exam header's levels line. Creation defaults: rung 2 Direct exercise · cue 1 Adjacent context · Theory. The API validates the exact unions — a float between rungs is a 400; legacy float-params sets fail the store validator and are skipped (no migration — `data/practice` is disposable scratch). -**Two-stage generation.** One **partitioner** agent reads the chains plus the three NAMED-LEVEL parameters and submits the true partition; each partition becomes one problem's target. Then **one generator agent per problem**: its prompt carries the problem-engineering framework verbatim (the owner's doctrine — the three distances, the search-to-solution axis, the rung ladder, reverse/discriminate/probe-the-premises, the standing prohibitions), **YOUR TARGET** (the assigned chains echoed verbatim — the echo law: the submit's chain set must equal the assignment exactly), and **THE FULL SET** (the whole selection: outside propositions are supporting material the generator may draw on with minimal use and maximal focus; the never-announce law covers ALL propositions). The submit is commit-before-pose — statement, solution, and rubric arrive together (the same sycophancy mitigation the quiz/explain mechanic uses); a problem never exists without its pre-written answer. Generators run in parallel (a process-wide 3-permit semaphore shared with grading) with per-problem failure isolation: one bad problem errors alone; the set still reaches `ready`. **Dedup across sets**: each generator also sees the problems of the ≤5 most recent sets sharing a node, so situations don't repeat between sets; within a set, parallel generators can't see each other — accepted for V1 (owner ruling). +**Two-stage generation.** One **partitioner** agent reads the chains plus the three NAMED-LEVEL parameters and submits the true partition; each partition becomes one problem's target. Then **one generator agent per problem**: its prompt carries the problem-engineering framework verbatim (the owner's doctrine — the three distances, the search-to-solution axis, the rung ladder, reverse/discriminate/probe-the-premises, the standing prohibitions, the formatting doctrine — full Markdown, LaTeX math, fenced code, tables/lists; prose-only renderings of structured content are a defect), **YOUR TARGET** (the assigned chains echoed verbatim — the echo law: the submit's chain set must equal the assignment exactly), and **THE FULL SET** (the whole selection: outside propositions are supporting material the generator may draw on with minimal use and maximal focus; the never-announce law covers ALL propositions). The submit is commit-before-pose — statement, solution, and rubric arrive together (the same sycophancy mitigation the quiz/explain mechanic uses); a problem never exists without its pre-written answer. Generators run in parallel (a process-wide 3-permit semaphore shared with grading) with per-problem failure isolation: one bad problem errors alone; the set still reaches `ready`. **Dedup across sets**: each generator also sees the problems of the ≤5 most recent sets sharing a node, so situations don't repeat between sets; within a set, parallel generators can't see each other — accepted for V1 (owner ruling). **Grading is feedback-only.** Submitting answers fires one **grader** job per answered problem: the grader sees the problem, the propositions its chains touch, the pre-written solution and rubric, and the learner's answer — never the full set (minimal, by ruling). It returns a verdict (`solid`/`partial`/`missed`) and feedback that names the failed recall and teaches the boundary. The verdict lands on the problem and the UI reveals it with the solution and rubric; the answer locks. Nothing reaches the graph, no level, no count, anywhere. A grader failure leaves the answer gradeless and re-submittable. diff --git a/src/agent/subagents/practice.ts b/src/agent/subagents/practice.ts index eaecccd..0ef6eb1 100644 --- a/src/agent/subagents/practice.ts +++ b/src/agent/subagents/practice.ts @@ -40,7 +40,7 @@ import { chainCoverage, CUE_LEVELS, LADDER_RUNGS, PRESENTATIONS } from "../../co // ─── doctrines ─────────────────────────────────────────────────────────────── -const GENERATOR_DOCTRINE = `# The problem engineering framework +export const GENERATOR_DOCTRINE = `# The problem engineering framework ## I. Theory @@ -188,6 +188,7 @@ INPUTS (in the user message): THE CUE DISTANCE: 0 Lesson context · 1 Adjacent context · 2 Far context — how far the situation sits from the cues the learner has already met: the lesson's own examples AND every previous problem's situation. Far context is the edge of the recognizable band — the proposition must still genuinely apply, and the learner must be able to find it by looking. THE PRESENTATION: Theory (the subject's own terms — symbols, formal objects, native vocabulary) or Application (a concrete worldly affair in which the proposition's structure appears in the wild). It dresses the situation; it never changes the rung or the cue distance. Apply Part II step 2: degrade honestly if the target cannot support them — the degradation shows in the problem's own shape, never in meta-commentary. +- FORMATTING: your statement, solution, and rubric are rendered as full Markdown — use everything it offers. Mathematics is always LaTeX: inline as $…$, display as $$…$$ on its own lines. Code, pseudocode, and formal systems live in fenced code blocks with a language tag; identifiers and symbols inline as \`code\`. Use tables for parallel structure and comparisons, lists for enumerations. Prose-only renderings of mathematical or structured content are a defect: if a sentence carries an equation, a derivation, or a piece of code, format it as what it is. - PREVIOUS PROBLEMS: problems the learner has already answered, with the chains they exercised. Do not rebuild one; keep your situation distant from theirs — cue distance counts previous problems' situations (Part I, "Cue distance"). OUTPUT — your entire deliverable is ONE submit_problem call: @@ -210,7 +211,7 @@ PARTITION DOCTRINE: OUTPUT — your entire deliverable is ONE submit_partition call: "partitions", a list of { chains: [ids] }, one entry per problem, together containing every input chain exactly once. No empty entries; no unknown ids; no problem without a real target relationship.`; -const GRADER_INSTRUCTIONS = `You are the grading stage of a learning application's practice system. One practice problem — built to strengthen the learner's recall of specific propositions — has been answered. Your job is FEEDBACK ONLY: verdict + what actually mattered. You are not assessing a person; you are coaching a recall. +export const GRADER_INSTRUCTIONS = `You are the grading stage of a learning application's practice system. One practice problem — built to strengthen the learner's recall of specific propositions — has been answered. Your job is FEEDBACK ONLY: verdict + what actually mattered. You are not assessing a person; you are coaching a recall. You receive: the problem's statement; its pre-written MODEL SOLUTION and GRADING RUBRIC (committed before the learner ever saw the problem — they are the reference, not you); the propositions (names + statements) the problem was built to exercise; and the learner's ANSWER — often dictated thinking-out-loud prose. Extract the reasoning from the prose; grade the substance, never the form. Do not demand the solution's exact wording — the rubric names what actually matters; nothing else is graded. @@ -227,6 +228,8 @@ THE VERDICT (exactly one): - "partial": some of the target came to mind; the answer is materially incomplete or partly wrong. - "missed": the targeted proposition(s) did not come to mind — the answer is empty or on the wrong track. +Feedback renders as full Markdown. Mathematics is always LaTeX — inline as $…$, display as $$…$$ on its own lines. Code in fenced blocks with a language tag, identifiers inline as \`code\`. Lists and tables wherever they carry structure. Name the failed recall and teach the boundary in properly formatted prose, not plain text. + OUTPUT — your entire deliverable is ONE submit_grade call: "verdict" and "feedback" (non-empty; the boundary-teaching text above).`; // ─── schemas + fills ───────────────────────────────────────────────────────── diff --git a/tests/practice-agents.test.ts b/tests/practice-agents.test.ts index 5446fc8..cb8cd3a 100644 --- a/tests/practice-agents.test.ts +++ b/tests/practice-agents.test.ts @@ -16,6 +16,10 @@ import { type ToolCall, } from "@earendil-works/pi-ai"; import type { PiEngine } from "../src/agent/models.ts"; +import { + GENERATOR_DOCTRINE, + GRADER_INSTRUCTIONS, +} from "../src/agent/subagents/practice.ts"; import { createPracticeAgents, renderChains, renderParams, renderPreviousProblems } from "../src/agent/subagents/practice.ts"; import { ModelsNotConfiguredError, ProviderKeyMissingError, type ProviderKeyMissing } from "../src/core/settings.ts"; import type { PracticeParams, ReasoningChain } from "../src/core/contracts.ts"; @@ -283,3 +287,40 @@ describe("refusal gates (BYOK honest refusal, extraction's pattern)", () => { agents.refuseIfStagesUnready(); // no throw }); }); + +// ─── formatting doctrine (owner-approved 2026-09-29) ───────────────────────── + +describe("formatting doctrine pins", () => { + it("GENERATOR_DOCTRINE carries the FORMATTING bullet verbatim (Markdown + LaTeX + fenced code + tables/lists)", () => { + assert.ok( + GENERATOR_DOCTRINE.includes( + "- FORMATTING: your statement, solution, and rubric are rendered as full Markdown — use everything it offers. Mathematics is always LaTeX: inline as $…$, display as $$…$$ on its own lines. Code, pseudocode, and formal systems live in fenced code blocks with a language tag; identifiers and symbols inline as `code`. Use tables for parallel structure and comparisons, lists for enumerations. Prose-only renderings of mathematical or structured content are a defect: if a sentence carries an equation, a derivation, or a piece of code, format it as what it is.", + ), + "the exact FORMATTING sentence", + ); + // it sits after the parameters bullet, before PREVIOUS PROBLEMS + assert.ok( + GENERATOR_DOCTRINE.indexOf("- FORMATTING:") > GENERATOR_DOCTRINE.indexOf("THE PRESENTATION:"), + "after the presentation bullet", + ); + assert.ok( + GENERATOR_DOCTRINE.indexOf("- PREVIOUS PROBLEMS:") > GENERATOR_DOCTRINE.indexOf("- FORMATTING:"), + "before PREVIOUS PROBLEMS", + ); + }); + + it("GRADER_INSTRUCTIONS carries the formatting paragraph before the OUTPUT section", () => { + assert.ok( + GRADER_INSTRUCTIONS.includes( + "Feedback renders as full Markdown. Mathematics is always LaTeX — inline as $…$, display as $$…$$ on its own lines. Code in fenced blocks with a language tag, identifiers inline as `code`. Lists and tables wherever they carry structure. Name the failed recall and teach the boundary in properly formatted prose, not plain text.", + ), + "the exact grader paragraph", + ); + assert.ok( + GRADER_INSTRUCTIONS.indexOf("Feedback renders as full Markdown") < GRADER_INSTRUCTIONS.indexOf("OUTPUT — your entire deliverable is ONE submit_grade call"), + "before the OUTPUT section", + ); + }); + + +});