From e6f52d3ad1c16dd044e2d97b0218e6718b89ecc7 Mon Sep 17 00:00:00 2001 From: amosblomqvist Date: Tue, 6 Oct 2026 07:34:51 +0300 Subject: [PATCH] docs: the assessment-levels vision (post-release arc) + the 0.3.0 release list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Vision-assessment records the 2026-10-06 thinking round: propositions as the unit, forecast semantics (mastery = P(correct unaided retrieval)), the Beta-with-decay estimator (BKT rejected: binary-only, guess-less, never forgets, unidentifiable at our volume; the weighted average completed with prior + recency + uncertainty), the e-table over outcome x rung x missClass, uniform 1/k credit (optional missed-field upgrade unruled), no new outcome enum values, PRACTICE-ONLY perimeter (ruled — quiz nodes are session-scoped plan objects, no honest mapping), mastery computed-on-read over the already durable problem/conclude calls (no stored ledger, graph untouched). Release-0.3 captures the owner's polish brain-dump (graph rendering toggle separate from extraction + open home-view question, question keybinds/VIM, session naming, markdown + graph exports, graph defaults, compaction dialog Esc law, keyboard audit) and the schema-lock position with the persistent surface inventory and pre-release gap list (schemaVersion stamps, pi-ai pins). --- docs/release-0.3.md | 64 +++++++++++++++++++++++++++++++ docs/vision-assessment.md | 81 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 145 insertions(+) create mode 100644 docs/release-0.3.md create mode 100644 docs/vision-assessment.md diff --git a/docs/release-0.3.md b/docs/release-0.3.md new file mode 100644 index 0000000..9a03206 --- /dev/null +++ b/docs/release-0.3.md @@ -0,0 +1,64 @@ +# First Release (0.3.0) — polish list + lock question + +Started 2026-10-06, owner brain-dump round. Priority 1 — everything else (assessment levels → docs/vision-assessment.md) comes after. Items are captured as spoken; design decisions inside them are **unruled** unless marked. + +## Polish items (owner list, 2026-10-06) + +1. **Graph rendering toggle — separate from the extraction pipeline.** + Today's graph on/off switch toggles extraction. Wanted: two switches — + one for the *pipeline* (named for what it does), one for *rendering* + ("graph on/off" = the graph view renders or not). Data, chains model, + extraction all keep working with rendering off. + - Open: **what is the home view when graph rendering is off?** It must + keep the import-notes button, the chains-view button, and the graph + pipeline cost readout. Candidate raised and not rejected: the chains + view (the chain table is already a complete non-graph presentation of + propositions + chains). A "giant DAG" was raised and rejected. Unruled. +2. **Keyboard navigation for questions** — arrow buttons (next/previous + question) are currently click-only; make them real keybinds, with VIM + bindings (j/k or similar) as the goal. Maybe rebindable, maybe not — unruled. +3. **Session naming.** First-message-as-name is not clean (the first message + is always long). Owner dislikes LLM-generated names. A naming popup after + first send was considered — possibly annoying; unruled. (Fact: the top-bar + title is already click-to-rename; whatever lands should agree with it.) +4. **Export sessions as markdown** — likely a right-click action on a session + in the sidebar. (Fact: core/session-markdown.ts already renders whole + sessions to markdown for extraction — the serialization exists.) +5. **Export the reasoning chains / graph** — format unruled (JSON is the + obvious candidate: serve the graph snapshot). Placement unruled. +6. **Graph default settings refinement** — current defaults not ideal; minor. +7. **Compaction summary dialog** — covers the entire screen and doesn't look + good; Esc prompts a dismiss question instead of just closing. Law stated: + **Esc always acts on the current element on screen.** +8. **General keyboard audit** — everything must work from the keyboard; some + minor things don't. Examples to be enumerated in the round. + +## What does "released" change in development? (the lock question) + +Position agreed 2026-10-06: releasing means **on-disk user data acquires +compatibility obligations** — after 0.3.0, persistent-schema changes are +migrations with tolerant reads, not free edits plus wipe-your-dev-data. +Nothing is frozen forever; the discipline changes. + +Persistent surfaces (the obligation set): +- `data/chats/*.json` — ChatRecord: messages (pi-ai wire shapes: thinking + blocks, toolCalls, usage), turns, kind, graphContext. The deepest lock — + and coupled to the pi-ai library's serialization (a pi-ai bump becomes a + migration event post-release). +- `data/graph/` — nodes.jsonl, chains.jsonl, coverage.jsonl, sources.json, + sessions.jsonl (append-only ledgers), positions.json, render-settings.json. +- `data/settings.json` (models, toggles, compaction…), `auth.json` (BYOK keys). +- Client localStorage (camera, filters, zoom, kind filter). +- `data/jobs/` — treat as scratch/transient, no obligation. + +Known gaps to close *before* release (proposals, unruled): +- **No schemaVersion anywhere.** Cheap now, load-bearing later: stamp + ChatRecord, settings, and the ledgers with a version at write time; + readers default missing → current. This is what makes later migrations + painless. +- **Pin pi-ai/pi-agent-core exact versions** so record shapes move only + deliberately. +- Release engineering outside code: signing/notarization await Apple + Developer enrollment (DMG is unsigned today); no auto-updater (manual + download acceptable for 0.3.0 if stated); packaged-app data location + already correct (LH_DATA_DIR → userData). diff --git a/docs/vision-assessment.md b/docs/vision-assessment.md new file mode 100644 index 0000000..749e839 --- /dev/null +++ b/docs/vision-assessment.md @@ -0,0 +1,81 @@ +# Vision — Assessment Levels (Mastery) + +Started 2026-10-06, thinking round (owner + assistant). **Deliberately unscheduled**: first release (0.3.0) comes first; this is the post-release arc. It resolves the "aggregation TBD" left open in vision-practice.md's assessment-record model (2026-09-27 rulings) — and inherits that round's law: *understanding is inferred, never encoded*. + +The surface for all of this is the future **data view** (still undesigned), precisely because these are computed estimates, not truths. Nothing here writes to the graph (the 2026-09-28 cleansing ruling stands: propositions + edges only, no understanding fields). + +## The unit: propositions (ruled 2026-10-06) + +Mastery is assessed **per proposition**. Not per chain, not per session, not per learner-skill. + +- The infrastructure already speaks propositions: practice problems tag target proposition ids, validated whole-graph at pose time (the round-9 mastery-trace contract, ids-first). +- The worry "knows all propositions ≠ knows the chain" is real but is an **instrument concern, not a model concern**: composition-mode problems *are* chain assessments (you cannot pass one without executing the inference), and their evidence already lands on member propositions. A chain whose propositions are all green but which never appears in a composition item is a problem-generation coverage gap. +- **Chain readiness is derived on read, never stored**: min (or product) over premise+conclusion mastery. For selection and display only. No second ledger. + +## What the number means: forecast semantics (ruled 2026-10-06) + +> **Mastery(p) = our current estimate of the probability that the learner can correctly exercise proposition p in a fresh, unaided retrieval.** + +We do not claim to observe understanding; we forecast performance. This makes the number honest (it is a probability, presented with its uncertainty, in the data view — never a verdict), naturally 0..1, and directly consumable: the doctrine's ~85%-success-per-retrieval target becomes selection arithmetic (pick propositions whose forecast sits in the ~0.85 band, decaying ones, or agent-flagged `retest`). + +It also gives the parked Brier idea its precise home: each new observation *scores* the pre-update forecast — (forecast − outcome)². "Are our numbers honest?" becomes measurable; a calibration panel is a natural data-view neighbor. + +Terminology: **mastery** is the display word. The precise phrase is *estimated recall reliability*; the definition carries the precision. + +## The estimator: a Beta belief with decay (agreed direction 2026-10-06) + +The owner's two candidates — BKT per proposition, or a weighted average of outcomes — resolve as: **BKT is the wrong tool; the weighted average is the right skeleton, completed with a prior, time decay, and uncertainty.** That completion makes it a Bayesian model anyway. + +Why BKT fails here: +- **Binary observations.** Our items are graded by the agent's rubric (outcome × rung × missClass — the richest signal we have); binarizing throws most of it away. +- **The guess parameter is fictitious** — the model sees the learner's thinking. Slip survives only as careless/execution misses, which missClass already *names* rather than smooths as noise. +- **Identifiability.** BKT wants p-init/p-learn/p-slip per skill; we will have ~1–10 observations per proposition. Per-proposition parameters cannot be fit; global parameters degenerate BKT into an EMA with ceremony. +- **BKT never forgets** (p-known is monotone). Mastery must decay with disuse. + +Three layers: + +**1 — the evidence scalar.** Each *concluded* problem becomes one number e ∈ [0,1]: "fraction of unaided mastery demonstrated." The agent's rubric, numerized — a tunable table, not physics: + +| outcome | e (approx.) | +|---|---| +| solved, no hint | ~0.95 (never 1.0 — one clean solve is not certainty) | +| solvedWithHints | ~0.85 → ~0.3 as rung climbs R1→R5 | +| movedOn | ~0.25 | +| surrendered | ~0.1 | +| dismissed | **excluded entirely** — nothing was demonstrated, in either direction | + +missClass modulates within a cell: *careless*/*execution* misses are failures not about the proposition (discount the negative evidence); *cue* and *structural* are the real negative signal; *discrimination* sits between. + +**2 — the belief.** Per proposition a Beta(α, β); each observation adds fractional success/failure counts weighted by: +- **target share 1/k** — uniform distribution across the problem's targets (correct, not just convenient: a k-target composition item genuinely spreads its evidence; counting it k full observations would inflate confidence k×); +- **recency** — exp(−Δt/τ), τ in weeks (forgetting; old evidence fades toward the prior); +- later, if the perimeter ever widens: an instrument weight. + +Posterior mean α/(α+β) is the displayed mastery; the spread is the honest error band (`0.62 ± wide · ~3 effective observations · last practiced 6d ago` — two data points visibly ≠ twenty). New propositions start at a weak prior (order Beta(2,5) ≈ 0.29 — propositions come from the learner's own material; low concentration so 2–3 observations move it decisively). Every knob (the e-table, τ, the prior) is a number in one place, tunable from field data without touching records or prompts. + +**3 — derived, never stored.** Chain readiness from member mastery (above). + +## Credit assignment (default ruled; the upgrade open) + +Default: uniform 1/k. The finer detail the owner wanted has a cheap shape, **unruled**: an optional `missed: [propositionIds]` field on the conclude call — the agent names which target(s) the miss was about, validated against the pose's own targets (round-9 discipline). Negative evidence then concentrates where it belongs; the other targets take fuller positive credit; "one third of a failure" becomes "the failure has an address." Rejected: per-submission grading (the conclude is the graded unit; grading every thinking message over-weights long problems and doubles agent work) and string-mining the verdict prose (drift). + +## Outcome levels: no new enum values (ruled 2026-10-06) + +The five-value outcome enum stays. outcome × rung × missClass already spans the space; felt missing resolution ("solved with a nudge" vs "solved after a walk-through") is carried by the rung. Resolution belongs in the scoring table (tunable in an afternoon); enum growth taxes every future conclude call — weak models, weak schemas (the GLM incident's lesson). + +## The perimeter: practice sessions only (ruled 2026-10-06) + +**Practice sessions are the sole source of assessment data.** Teaching sessions never contribute, even though they contain proposition-shaped evidence (quizzes, explain free-recall, predict/check). Reasons: clear boundaries and a single truth per construct; teaching is an entirely different purpose (active, just-taught material vs cold retrieval); and concretely, the nodes quiz questions reference are session-scoped plan objects, *not* graph propositions — no honest mapping exists (the 2026-09-27 guardrail). The evidence record still carries a `source` field internally, but only practice writes it. + +## Storage: nothing new, on principle (ruled 2026-10-06) + +The raw evidence is already durable: problem calls carry targets + mode; conclude calls carry outcome + rung + missClass; both live in append-only chat records under honest activation. **The assessment trace is a join over what exists; mastery is a computed view** (like graphSnapshot) — cached only if it ever gets slow. If the practice agent later needs mastery as a selection input, it reads the computed view at session start. No stored ledger, no drift with the records, graph untouched. + +## Open questions (unruled) + +- The prior's center: Beta(2,5)-ish, or higher for propositions from the learner's own documents? +- The `missed` field on conclude: adopt or not? +- τ (the decay half-life): weeks — set from field data once it exists. +- The e-table's exact cells: calibrate after the first real corpus of concluded problems. +- Whether mastery ever feeds the practice agent's *live* problem selection during a session (the arc's second half — adaptive selection) or only at session start. +- The data view's presentation of all of this (its own design round).