diff --git a/README.md b/README.md index 7d50988..bd09a47 100644 --- a/README.md +++ b/README.md @@ -1,178 +1,47 @@ -# learning-harness +# LearnHarness -A standalone learning app (web + Electron desktop): a **teacher agent** (Next.js; the Vercel AI SDK's `UIMessage` protocol end-to-end, LLM engine [pi-ai](docs/pi-engine-swap.md) via OpenRouter), probe→teach lessons with graded quiz/ask interactions, a formal **session plan tree** the teacher authors once (`plan_init`) and whose progress is derived from tagged quiz calls, and a persistent cross-session **knowledge graph** of what the learner knows — the graph is the app's home page. +A desktop AI tutor that teaches by questioning — one reasoning step at a time — and keeps everything you learn in a growing knowledge graph. -Not a chat app with a graph bolted on: sessions are probe→teach lessons, the plan tree is a structured, schema-validated object, sessions accumulate into the graph through the extraction pipeline (propositions + reasoning chains — the stored truth; the canvas renders a derived image), and the teacher reaches back into the graph through read-only prober tools on its own judgment so probing compounds across sessions. +It runs on your machine, with your own API key. Your data stays in plain files you can open. -- [`docs/architecture.md`](docs/architecture.md) is the **canonical architecture spec** — layers, the import law, the five systems, the invariants, the seam directory. (The pre-audit spec is archived under `docs/archive/` — history only.) -- Living design doc (all decisions): [`docs/vision.md`](docs/vision.md) -- Build plan + wave/gate results: [`docs/plan.md`](docs/plan.md) +## The idea -## Commands +Understanding has a dependency structure: some things require other things first. A lesson that ignores what you already hold is wasted on you; a lesson that outruns what you hold is lost on you. There is exactly one optimal path — the smallest set of steps from what you know to what you want to know — and finding it requires knowing precisely what you know. + +So LearnHarness measures first. It questions you to bracket the edge of your understanding, plans the shortest path from that edge to your goal, then teaches it one reasoning step at a time — checking that each step lands before building on it. Every step is small enough to feel obvious on arrival. + +And it remembers. Everything taught accumulates into a single persistent structure — propositions and the reasoning chains that connect them — rendered as the graph that is the app's home view. The next session starts from what the last one built: no re-probing what you already hold, questions aimed at the actual boundary of your knowledge, learning that compounds. + +Teaching installs understanding; practice makes it available when you need it. Separate practice sessions draw problems from the same graph — retrieval at the edge of your ability, not review — because being taught something and having it come to mind in the right situation are different states. + +## Yours, locally + +A system whose premise is that it can be trusted is built so that it can be checked: + +- **Your keys.** Bring your own ([OpenRouter](https://openrouter.ai/) works out of the box). No subscription, no markup — a live cost meter shows what each session actually spent. +- **Your machine.** Chats, graph, settings — plain files under your home directory. Nothing is sent anywhere except the model calls you configure. +- **Inspectable.** Every system prompt is visible and tunable. The graph is stored as reasoning chains — the structure of what you were taught, not a pile of text. + +## Download + +**[Releases](https://github.com/amosblomqvist/learnharness/releases)** — macOS on Apple Silicon, signed and notarized. Windows and Linux are planned. + +## Quick start + +1. Download the DMG and drag the app to Applications. +2. Open Settings, paste your OpenRouter API key, and pick your models. +3. Start a session and say what you want to understand. + +## Development ```bash -npm install # once - -npm run dev # http://localhost:3000 -npm run build # production build (standalone output) -npm start # serve the production build - -npm test # all 44 suites (node:test via tsx; deterministic, no LLM) -npm run lint # eslint — includes the import law (architecture §2) -npm run typecheck # tsc --noEmit - -npm run desktop:dev # next dev + Electron shell pointed at it -npm run desktop:dist # build + package the desktop app (electron-builder) - -# one-shot data scripts (run from the master tree only — data never crosses trees) -npx tsx --env-file=.env.local scripts/import-notes.ts # bulk-import notes → graph -npx tsx --env-file=.env.local scripts/backfill-graph.ts # wipe + re-extract every chat -npx tsx --env-file=.env.local scripts/replay-layout.ts # re-derive layout from the ledgers -npx tsx scripts/sync-pricing.ts # regenerate core/prices.ts tables +npm install +npm run dev # http://localhost:3000 +npm test # deterministic, no LLM, no network ``` -## Environment (`.env.local`) +The architecture spec and full development guide live in [`docs/`](docs/) — start with [`docs/architecture.md`](docs/architecture.md) and [`docs/development.md`](docs/development.md). -Secrets and machine-specific paths only. Model choice and thinking budgets are the user's, not the environment's — BYOK model slots live in **settings** (`models` section: strong/cheap/vision, provider + model id + optional thinking level; per-role and per-chat overrides ride on top). The extraction batch threshold is a settings knob too (`graph.extractionChunkTokens`, default 4096, floor 500). +## License -| Variable | Purpose | -|---|---| -| `OPENROUTER_API_KEY` | required — all LLM calls go through OpenRouter | -| `GOOGLE_SEARCH_API_KEY` + `GOOGLE_CSE_ID` | researcher's `web_search` (Google Custom Search) | -| `DEEPGRAM_API_KEY` | optional — dictation (STT token minting; audio streams browser→Deepgram, never through the server) | -| `CHROME_PATH` | optional explicit Chrome path for headless verification (puppeteer-core uses system Chrome; nothing is downloaded) | - -The packaged desktop app takes keys from `userData/auth.json` (BYOK, mode 0600, injected into the server child as env only). - -## Architecture - -Five layers, dependency arrows pointing inward only, machine-enforced as four ESLint zones (shell → barrels only; core and agent never import below themselves). Two composition roots, one writer per store. The full spec — layers, systems, runtime flows, invariants, seams: [`docs/architecture.md`](docs/architecture.md). - -```mermaid -flowchart TB - LEARNER["the learner"] - - subgraph STAGE["STAGE — presentation shell (browser)"] - UI["session UI · question dock · plan panel · graph canvas · activity dock · sidebar"] - end - - subgraph CLASSROOM["CLASSROOM — session engine (owns data/chats)"] - TURN["turn pipeline — handleChatRequest:
liveness · steer claiming · write-ahead persist · stream"] - AGENTS["prober → teacher agents
(stateless per-turn, built from the record)"] - CHATDATA[("chat records
data/chats/*.json")] - end - - subgraph WORKBENCH["WORKBENCH — background labor (owns data/jobs)"] - SUBS["researcher · demo-builder · visual-maker
(pi agents driven to completion)"] - WAKE["wake pipeline — per-chat serialized steer delivery,
server-side wake turns, crash recovery"] - end - - subgraph LIBRARY["LIBRARY — world model (owns data/graph ledgers)"] - EXTRACT["extraction pipeline:
extractor → auditor → merger
(the SOLE graph writer, serial per source)"] - DERIVE["derivation — graphSnapshot · birth-session clusters
(pure reads)"] - LEDG[("append-only ledgers
data/graph/*.jsonl")] - end - - subgraph ROOM["MACHINE ROOM — host"] - HOST["Electron main → standalone Next server ·
config.ts · LH_DATA_DIR · BYOK keys"] - end - - LEARNER -->|"text · graded answers"| UI - UI -->|"POST /api/chat + addToolOutput"| TURN - TURN -->|"quiz / ask stream to browser
(client tools, no server execute)"| UI - TURN --> CHATDATA - TURN -->|"spawn(job) — fire-and-forget ack"| SUBS - SUBS -->|"result → steer message"| WAKE - WAKE -->|"exactly-once wake turn"| TURN - TURN -->|"trigger when uncovered tokens ≥ chunkTokens"| EXTRACT - EXTRACT -->|"reads transcript as evidence"| CHATDATA - EXTRACT --> LEDG - EXTRACT -->|"semantic top-K context injection (one-shot steer)"| TURN - LEDG --> DERIVE - UI -->|"GET /api/graph · PUT positions · POST import"| DERIVE - HOST -.->|"hosts the server side"| TURN -``` - -**Core invariant:** the extraction pipeline (extractor → auditor → merger) is the sole graph writer — tree-blind and post-hoc, triggered by token coverage, not quiz counts. The LLM stages propose, verify, and judge identity (link-or-create against the current graph; each finding carries verbatim evidence and message-granular provenance); the deterministic merger owns every ledger append (ids, provenance, chain dedupe — the same argument re-asserted merges under one `arg_` id). Every node is an objective, world-facing fact; the learner's relationship to it lives entirely in practice records — feedback-only, never written back. The teacher stays a teacher; everything below the ledgers is derived. - -## Layout - -``` -app/ Next.js App Router — pages + 16 thin API route handlers -components/ session + home UI, question dock, plan panel, graph - canvas + view, artifact renderers (registry: artifacts/), - dictation, activity dock, keyboard machine -src/ - agent/ - system-prompt.md base teaching behavior (the methodology) - prompts/{prober,teacher}/ composed numbered fragments per role - teacher-agent.ts prober/teacher factories (engine injectable for tests) - tools.ts tool composition; quiz / ask_user_question (client tools) - tools/plan.ts plan_init (author-once mermaid plan, deterministic parser) - tools/probe.ts probe_done (probe→teach handoff seam) - tools/jobs.ts research / demo / visual tool surfaces - tools/web.ts web_search (Google CSE) + web_fetch (researcher only) - subagents/ researcher · demo-builder · visual-maker - · pdf-reader, - extraction pipeline (extractor/auditor + merger handoff), - runner (async protocol), headless Chrome helper, - abort registry - server/chat-handler.ts THE turn composition root (liveness · steers · persist · stream) - server/turn-service.ts turn seams: persist · extraction trigger - server/jobs.ts the wake pipeline: serialized per-chat steer delivery, - server-side wake turns, crash recovery - server/stores.ts THE storage composition root — the one place adapters wire in - server/turn-liveness.ts per-chat turn registry (one turn per chat, ever) - server/observability.ts the activity feed (running jobs + extraction, abort) - server/pi/ the engine bridge: record-adapter (record → pi messages) - · ui-stream (pi events → UIMessage SSE) - server/index.ts the server barrel (the only server import app/ may use) - core/contracts/{plan,graph,chat,jobs,probe,models}.ts - the contracts — one file per domain - core/ports.ts storage ports — ChatStore · GraphStore · JobStore - (core code depends on these types only) - core/ question-core, plan-tree, phases, pending, graph - (merger/derivation), coverage, layout, clusters, - prices, tool-parts — all pure: storage - arrives as parameters - store/ fs adapters implementing the ports: chats · graph - ledgers · jobs (+ render-settings, default-settings) -electron/ desktop launcher + security guard (spawns the standalone - server; 1 IPC channel; LH_DATA_DIR + BYOK env) -scripts/ build-server.mjs (packaging guards), import-notes, - backfill-graph, replay-layout, sync-pricing -data/ chats/ · graph/ (nodes/chains/sessions/coverage - ledgers + positions/sources/render-settings maps) · - jobs/ · settings.json — runtime state, git-ignored -``` - -## Key mechanisms - -**Client-side question tools.** `quiz` and `ask_user_question` have no server `execute`: the tool call streams to the browser, the learner answers in the dock, the client grades via `question-core` and returns the verdict with `addToolOutput` — which fires the SDK continuation so the teacher resumes mid-turn. Options are shuffled deterministically (seeded by toolCallId) so server and client renders match. A pending quiz persisted in the chat record is exactly what makes **resume mid-question free**. - -**The formal plan tree (author-once).** The teacher authors the session's dependency tree with one schema-validated tool — `plan_init`, whose mermaid-subset input is parsed deterministically (line-numbered errors) and validated as a DAG. The tree persists in the chat record; progress is never mutated — `current`/`visited`/upcoming are **derived** from tagged quiz calls in the transcript, and the panel renders that derivation. - -**The async sub-agent protocol (ack-and-steer).** `research` / `demo` / `visual` are server-executed tools: the call resolves immediately mid-turn with a fire-and-forget ack and the teacher keeps teaching; a background runner works in parallel (the client polls `/api/activity` and `/api/jobs`). On completion the runner builds a **steer message** — a user-role `UIMessage` whose text part is the model-facing result and whose `data-subagent-artifact` part carries the full result for the client — and the **wake pipeline** delivers it: a per-chat serialized queue waits for the turn to end and no question to be pending, then fires a server-side teacher turn that weaves the result into the lesson. Delivery is exactly-once (deterministic steer ids, delivered-flag checks at every claim point); stalled jobs are recovered on read; aborted jobs deliver an abort steer. The client adopts the wake turn through the same `turnInFlight` machinery as returning mid-stream (busy shimmer, composer gate) and renders steers as artifacts — never a bubble. Sub-agents **verify, never one-shot**: the demo-builder headlessly executes its HTML in system Chrome before delivery; the visual-maker renders each SVG and looks at it (vision-capable model — the `vision` slot in settings). - -**The knowledge graph (two channels, one writer).** The write channel: the extraction pipeline — extractor (LLM: sees the source — the session's teacher prose or a notes chunk — plus its own source's digest (**THE GRAPH SO FAR**: propositions + chains already extracted from this source, ids verbatim — digest-aware redesign, 2026-09-30 evening: the extractor IS the assembler, referencing existing ids instead of restating, leaning on established conclusions as premises); fully trusted; its deliverable is core ideas + reasoning chains: premises that jointly entail a conclusion, submitted as chains and stored as chains — `chains.jsonl` is the truth, edges a derived view; owner redesign + chains-primary, 2026-09-30) → auditor (LLM, an agentic loop that never sees the source: explores the existing graph through `find` / `list_sessions` / `list_propositions` tools and stamps each proposition with an identity verdict — link to an existing node (equivalence bias: paraphrases merge) or create) → merger (`mergeFindings`, deterministic: applies the verdicts, link-appends phrasings/provenance, resolves and dedupes chains — identical arguments merge under one `arg_` id; then the chain-simplification pass strips provably redundant derivations — prefer-fewest-premises + ancestry-cover — checkpoints coverage) → layout settle. A source's batches run strictly in order (each extractor continues the digest of every earlier merge). No quiz outcomes or understanding assessments live in the graph (2026-09-27 cleanup); practice (`/practice`) consumes the graph as feedback-only problem sets and writes nothing back. Triggered when a chat's **uncovered tokens** (message-view fingerprints, `coverage.jsonl`) cross `graph.extractionChunkTokens` (a settings knob, default 8192, floor 500 — read fresh per turn); failure checkpoints nothing, the next trigger re-runs. The same pipeline serves the **notes front door** (`POST /api/graph/import`: any `*.md`/`*.txt` becomes a source; re-imports are idempotent by path hash). The read channel: `GET /api/graph` derives the snapshot from the ledgers — a pure read (`deriveView` rebuilds the lossy image: each chain an implication node `⟹` with premise→implication→conclusion arrows, so two arguments for one conclusion stay two diamonds); the home view is a hand-rolled Canvas 2D renderer (one rAF loop, dirty-flag idle, d3-zoom camera, zoom-gated labels, cluster hulls) over a d3-force layout, with server-persisted render settings and drag-committed positions. Clusters are **birth-session groupings**: a node joins the cluster of the session that birthed it (a chat, or an imported notes file as a file-kind session), derived at read — no embedding pass, no naming pass. The teacher reaches the graph on its own terms through the gated prober tools (`list_sessions` / `list_propositions`, the `agent.graphTools` setting) — there is no automatic context injection. - -**Practice (problem sets).** A node selection resolves to the STORED reasoning chains it fully contains (a chain rides iff every member is selected; unchained nodes ride as singletons); the `/practice` creation dialog picks a birth-session cluster (the graph view's own groups) and freezes its propositions + chains into a self-contained `ProblemSet`. Three named-level axes steer generation (ladder rung 0–4, cue distance 0–2, Theory|Application — never a number to interpolate; defaults Direct exercise · Adjacent context · Theory), plus an optional free-form note the agents weigh but need not follow. Creation has two entries: the cluster picker on `/practice`, or the graph view's chain table — a full-bleed sortable table of every reasoning chain, grouped by birth cluster under the shared top bar, selecting chips, chains, or whole clusters, whose floating selection bar shows the exact `N nodes → M chains` the agents will receive — handing a free-form node selection to the same dialog — and carries the fenced Delete action, making it the graph's management view (whole-cluster cleanup through the same full-death confirm dialog). A partitioner agent covers every chain exactly once, one generator per partition writes problems against the owner's problem-engineering framework (commit-before-pose: statement + solution + rubric together), and grading is **strengthening, not assessment** — one attempt per problem with its own submit button, instant solution/rubric reveal, then a score 0–10 with brief reasoning landing live; nothing ever written back to the graph. Runs are registry-backed background jobs on the activity dock (no steer, never wakes the teacher); sets live in `data/practice`, crash-swept on boot. - -**Dictation.** Optional speech-to-text: the server mints short-lived Deepgram tokens; audio streams browser→Deepgram over WebSocket, never through the server. - -## Testing - -All deterministic, no LLM, no network (engines are injected — pi-ai's faux provider scripted per test; `globalThis.fetch` stubbed where transport matters): - -1. **Pure core units** — merger (via the real fs adapters, backup-and-restored), derivation, coverage, question grading, plan tree, phases, clusters, layout, tool-parts, prices. -2. **Server flows** — `handleChatRequest` driven end-to-end with scripted engines: persistence, liveness, steer claiming, the wake pipeline (409 retry, quiz deferral, idempotency), crash recovery, the data-dir seam (including a fresh child process). -3. **Sub-agent loops** — researcher/demo-builder/visual-maker with fake headless APIs; the extraction pipeline end-to-end (audit drops, failure-checkpoints-nothing, single-flight, abort). -4. **Render tests** — transcript, cards, docks, composer, sidebar, coverage via React server-side rendering. - -## Lineage - -The teaching methodology comes from the Obsidian-based POC (`~/Desktop/Obsidian/vault/π/learn`; its `knowledge graph design.md` was the original epistemic-model source, since amended — see the amendments table in `docs/vision.md`). The harness was originally built on the pi agent SDK, pivoted to standalone Next.js + the Vercel AI SDK, and later swapped its LLM engine back to pi-ai while keeping the AI SDK's `UIMessage` as the client protocol ([`docs/pi-engine-swap.md`](docs/pi-engine-swap.md)). - -## Data - -Nothing under `data/` is ever tracked by git. Chats, graph ledgers, jobs, positions, and the settings records are plain files (JSON/JSONL) — inspect them freely; they are the canonical records. The desktop app relocates the whole directory via `LH_DATA_DIR`. \ No newline at end of file +[AGPL-3.0](LICENSE) diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..e89f3d0 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,180 @@ +# Development guide + +The developer README (split out of the storefront README, 2026-10-10). Everything below is for people working ON LearnHarness, not using it. + +A standalone learning app (web + Electron desktop): a **teacher agent** (Next.js; the Vercel AI SDK's `UIMessage` protocol end-to-end, LLM engine [pi-ai](docs/pi-engine-swap.md) via OpenRouter), probe→teach lessons with graded quiz/ask interactions, a formal **session plan tree** the teacher authors once (`plan_init`) and whose progress is derived from tagged quiz calls, and a persistent cross-session **knowledge graph** of what the learner knows — the graph is the app's home page. + +Not a chat app with a graph bolted on: sessions are probe→teach lessons, the plan tree is a structured, schema-validated object, sessions accumulate into the graph through the extraction pipeline (propositions + reasoning chains — the stored truth; the canvas renders a derived image), and the teacher reaches back into the graph through read-only prober tools on its own judgment so probing compounds across sessions. + +- [`docs/architecture.md`](docs/architecture.md) is the **canonical architecture spec** — layers, the import law, the five systems, the invariants, the seam directory. (The pre-audit spec is archived under `docs/archive/` — history only.) +- Living design doc (all decisions): [`docs/vision.md`](docs/vision.md) +- Build plan + wave/gate results: [`docs/plan.md`](docs/plan.md) + +## Commands + +```bash +npm install # once + +npm run dev # http://localhost:3000 +npm run build # production build (standalone output) +npm start # serve the production build + +npm test # the full suite (node:test via tsx; deterministic, no LLM, no network) +npm run lint # eslint — includes the import law (architecture §2) +npm run typecheck # tsc --noEmit + +npm run desktop:dev # next dev + Electron shell pointed at it +npm run desktop:dist # build + package the desktop app (electron-builder) + +# one-shot data scripts (run from the master tree only — data never crosses trees) +npx tsx --env-file=.env.local scripts/import-notes.ts # bulk-import notes → graph +npx tsx --env-file=.env.local scripts/backfill-graph.ts # wipe + re-extract every chat +npx tsx --env-file=.env.local scripts/replay-layout.ts # re-derive layout from the ledgers +npx tsx scripts/sync-pricing.ts # regenerate core/prices.ts tables +``` + +## Environment (`.env.local`) + +Secrets and machine-specific paths only. Model choice and thinking budgets are the user's, not the environment's — BYOK model slots live in **settings** (`models` section: strong/cheap/vision, provider + model id + optional thinking level; per-role and per-chat overrides ride on top). The extraction batch threshold is a settings knob too (`graph.extractionChunkTokens`, default 4096, floor 500). + +| Variable | Purpose | +|---|---| +| `OPENROUTER_API_KEY` | required — all LLM calls go through OpenRouter | +| `GOOGLE_SEARCH_API_KEY` + `GOOGLE_CSE_ID` | researcher's `web_search` (Google Custom Search) | +| `DEEPGRAM_API_KEY` | optional — dictation (STT token minting; audio streams browser→Deepgram, never through the server) | +| `CHROME_PATH` | optional explicit Chrome path for headless verification (puppeteer-core uses system Chrome; nothing is downloaded) | + +The packaged desktop app takes keys from `userData/auth.json` (BYOK, mode 0600, injected into the server child as env only). + +## Architecture + +Five layers, dependency arrows pointing inward only, machine-enforced as four ESLint zones (shell → barrels only; core and agent never import below themselves). Two composition roots, one writer per store. The full spec — layers, systems, runtime flows, invariants, seams: [`docs/architecture.md`](docs/architecture.md). + +```mermaid +flowchart TB + LEARNER["the learner"] + + subgraph STAGE["STAGE — presentation shell (browser)"] + UI["session UI · question dock · plan panel · graph canvas · activity dock · sidebar"] + end + + subgraph CLASSROOM["CLASSROOM — session engine (owns data/chats)"] + TURN["turn pipeline — handleChatRequest:
liveness · steer claiming · write-ahead persist · stream"] + AGENTS["prober → teacher agents
(stateless per-turn, built from the record)"] + CHATDATA[("chat records
data/chats/*.json")] + end + + subgraph WORKBENCH["WORKBENCH — background labor (owns data/jobs)"] + SUBS["researcher · demo-builder · visual-maker
(pi agents driven to completion)"] + WAKE["wake pipeline — per-chat serialized steer delivery,
server-side wake turns, crash recovery"] + end + + subgraph LIBRARY["LIBRARY — world model (owns data/graph ledgers)"] + EXTRACT["extraction pipeline:
extractor → auditor → merger
(the SOLE graph writer, serial per source)"] + DERIVE["derivation — graphSnapshot · birth-session clusters
(pure reads)"] + LEDG[("append-only ledgers
data/graph/*.jsonl")] + end + + subgraph ROOM["MACHINE ROOM — host"] + HOST["Electron main → standalone Next server ·
config.ts · LH_DATA_DIR · BYOK keys"] + end + + LEARNER -->|"text · graded answers"| UI + UI -->|"POST /api/chat + addToolOutput"| TURN + TURN -->|"quiz / ask stream to browser
(client tools, no server execute)"| UI + TURN --> CHATDATA + TURN -->|"spawn(job) — fire-and-forget ack"| SUBS + SUBS -->|"result → steer message"| WAKE + WAKE -->|"exactly-once wake turn"| TURN + TURN -->|"trigger when uncovered tokens ≥ chunkTokens"| EXTRACT + EXTRACT -->|"reads transcript as evidence"| CHATDATA + EXTRACT --> LEDG + EXTRACT -->|"semantic top-K context injection (one-shot steer)"| TURN + LEDG --> DERIVE + UI -->|"GET /api/graph · PUT positions · POST import"| DERIVE + HOST -.->|"hosts the server side"| TURN +``` + +**Core invariant:** the extraction pipeline (extractor → auditor → merger) is the sole graph writer — tree-blind and post-hoc, triggered by token coverage, not quiz counts. The LLM stages propose, verify, and judge identity (link-or-create against the current graph; each finding carries verbatim evidence and message-granular provenance); the deterministic merger owns every ledger append (ids, provenance, chain dedupe — the same argument re-asserted merges under one `arg_` id). Every node is an objective, world-facing fact; the learner's relationship to it lives entirely in practice records — feedback-only, never written back. The teacher stays a teacher; everything below the ledgers is derived. + +## Layout + +``` +app/ Next.js App Router — pages + 16 thin API route handlers +components/ session + home UI, question dock, plan panel, graph + canvas + view, artifact renderers (registry: artifacts/), + dictation, activity dock, keyboard machine +src/ + agent/ + system-prompt.md base teaching behavior (the methodology) + prompts/{prober,teacher}/ composed numbered fragments per role + teacher-agent.ts prober/teacher factories (engine injectable for tests) + tools.ts tool composition; quiz / ask_user_question (client tools) + tools/plan.ts plan_init (author-once mermaid plan, deterministic parser) + tools/probe.ts probe_done (probe→teach handoff seam) + tools/jobs.ts research / demo / visual tool surfaces + tools/web.ts web_search (Google CSE) + web_fetch (researcher only) + subagents/ researcher · demo-builder · visual-maker + · pdf-reader, + extraction pipeline (extractor/auditor + merger handoff), + runner (async protocol), headless Chrome helper, + abort registry + server/chat-handler.ts THE turn composition root (liveness · steers · persist · stream) + server/turn-service.ts turn seams: persist · extraction trigger + server/jobs.ts the wake pipeline: serialized per-chat steer delivery, + server-side wake turns, crash recovery + server/stores.ts THE storage composition root — the one place adapters wire in + server/turn-liveness.ts per-chat turn registry (one turn per chat, ever) + server/observability.ts the activity feed (running jobs + extraction, abort) + server/pi/ the engine bridge: record-adapter (record → pi messages) + · ui-stream (pi events → UIMessage SSE) + server/index.ts the server barrel (the only server import app/ may use) + core/contracts/{plan,graph,chat,jobs,probe,models}.ts + the contracts — one file per domain + core/ports.ts storage ports — ChatStore · GraphStore · JobStore + (core code depends on these types only) + core/ question-core, plan-tree, phases, pending, graph + (merger/derivation), coverage, layout, clusters, + prices, tool-parts — all pure: storage + arrives as parameters + store/ fs adapters implementing the ports: chats · graph + ledgers · jobs (+ render-settings, default-settings) +electron/ desktop launcher + security guard (spawns the standalone + server; 1 IPC channel; LH_DATA_DIR + BYOK env) +scripts/ build-server.mjs (packaging guards), import-notes, + backfill-graph, replay-layout, sync-pricing +data/ chats/ · graph/ (nodes/chains/sessions/coverage + ledgers + positions/sources/render-settings maps) · + jobs/ · settings.json — runtime state, git-ignored +``` + +## Key mechanisms + +**Client-side question tools.** `quiz` and `ask_user_question` have no server `execute`: the tool call streams to the browser, the learner answers in the dock, the client grades via `question-core` and returns the verdict with `addToolOutput` — which fires the SDK continuation so the teacher resumes mid-turn. Options are shuffled deterministically (seeded by toolCallId) so server and client renders match. A pending quiz persisted in the chat record is exactly what makes **resume mid-question free**. + +**The formal plan tree (author-once).** The teacher authors the session's dependency tree with one schema-validated tool — `plan_init`, whose mermaid-subset input is parsed deterministically (line-numbered errors) and validated as a DAG. The tree persists in the chat record; progress is never mutated — `current`/`visited`/upcoming are **derived** from tagged quiz calls in the transcript, and the panel renders that derivation. + +**The async sub-agent protocol (ack-and-steer).** `research` / `demo` / `visual` are server-executed tools: the call resolves immediately mid-turn with a fire-and-forget ack and the teacher keeps teaching; a background runner works in parallel (the client polls `/api/activity` and `/api/jobs`). On completion the runner builds a **steer message** — a user-role `UIMessage` whose text part is the model-facing result and whose `data-subagent-artifact` part carries the full result for the client — and the **wake pipeline** delivers it: a per-chat serialized queue waits for the turn to end and no question to be pending, then fires a server-side teacher turn that weaves the result into the lesson. Delivery is exactly-once (deterministic steer ids, delivered-flag checks at every claim point); stalled jobs are recovered on read; aborted jobs deliver an abort steer. The client adopts the wake turn through the same `turnInFlight` machinery as returning mid-stream (busy shimmer, composer gate) and renders steers as artifacts — never a bubble. Sub-agents **verify, never one-shot**: the demo-builder headlessly executes its HTML in system Chrome before delivery; the visual-maker renders each SVG and looks at it (vision-capable model — the `vision` slot in settings). + +**The knowledge graph (two channels, one writer).** The write channel: the extraction pipeline — extractor (LLM: sees the source — the session's teacher prose or a notes chunk — plus its own source's digest (**THE GRAPH SO FAR**: propositions + chains already extracted from this source, ids verbatim — digest-aware redesign, 2026-09-30 evening: the extractor IS the assembler, referencing existing ids instead of restating, leaning on established conclusions as premises); fully trusted; its deliverable is core ideas + reasoning chains: premises that jointly entail a conclusion, submitted as chains and stored as chains — `chains.jsonl` is the truth, edges a derived view; owner redesign + chains-primary, 2026-09-30) → auditor (LLM, an agentic loop that never sees the source: explores the existing graph through `find` / `list_sessions` / `list_propositions` tools and stamps each proposition with an identity verdict — link to an existing node (equivalence bias: paraphrases merge) or create) → merger (`mergeFindings`, deterministic: applies the verdicts, link-appends phrasings/provenance, resolves and dedupes chains — identical arguments merge under one `arg_` id; then the chain-simplification pass strips provably redundant derivations — prefer-fewest-premises + ancestry-cover — checkpoints coverage) → layout settle. A source's batches run strictly in order (each extractor continues the digest of every earlier merge). No quiz outcomes or understanding assessments live in the graph (2026-09-27 cleanup); practice (`/practice`) consumes the graph as feedback-only problem sets and writes nothing back. Triggered when a chat's **uncovered tokens** (message-view fingerprints, `coverage.jsonl`) cross `graph.extractionChunkTokens` (a settings knob, default 8192, floor 500 — read fresh per turn); failure checkpoints nothing, the next trigger re-runs. The same pipeline serves the **notes front door** (`POST /api/graph/import`: any `*.md`/`*.txt` becomes a source; re-imports are idempotent by path hash). The read channel: `GET /api/graph` derives the snapshot from the ledgers — a pure read (`deriveView` rebuilds the lossy image: each chain an implication node `⟹` with premise→implication→conclusion arrows, so two arguments for one conclusion stay two diamonds); the home view is a hand-rolled Canvas 2D renderer (one rAF loop, dirty-flag idle, d3-zoom camera, zoom-gated labels, cluster hulls) over a d3-force layout, with server-persisted render settings and drag-committed positions. Clusters are **birth-session groupings**: a node joins the cluster of the session that birthed it (a chat, or an imported notes file as a file-kind session), derived at read — no embedding pass, no naming pass. The teacher reaches the graph on its own terms through the gated prober tools (`list_sessions` / `list_propositions`, the `agent.graphTools` setting) — there is no automatic context injection. + +**Practice (problem sets).** A node selection resolves to the STORED reasoning chains it fully contains (a chain rides iff every member is selected; unchained nodes ride as singletons); the `/practice` creation dialog picks a birth-session cluster (the graph view's own groups) and freezes its propositions + chains into a self-contained `ProblemSet`. Three named-level axes steer generation (ladder rung 0–4, cue distance 0–2, Theory|Application — never a number to interpolate; defaults Direct exercise · Adjacent context · Theory), plus an optional free-form note the agents weigh but need not follow. Creation has two entries: the cluster picker on `/practice`, or the graph view's chain table — a full-bleed sortable table of every reasoning chain, grouped by birth cluster under the shared top bar, selecting chips, chains, or whole clusters, whose floating selection bar shows the exact `N nodes → M chains` the agents will receive — handing a free-form node selection to the same dialog — and carries the fenced Delete action, making it the graph's management view (whole-cluster cleanup through the same full-death confirm dialog). A partitioner agent covers every chain exactly once, one generator per partition writes problems against the owner's problem-engineering framework (commit-before-pose: statement + solution + rubric together), and grading is **strengthening, not assessment** — one attempt per problem with its own submit button, instant solution/rubric reveal, then a score 0–10 with brief reasoning landing live; nothing ever written back to the graph. Runs are registry-backed background jobs on the activity dock (no steer, never wakes the teacher); sets live in `data/practice`, crash-swept on boot. + +**Dictation.** Optional speech-to-text: the server mints short-lived Deepgram tokens; audio streams browser→Deepgram over WebSocket, never through the server. + +## Testing + +All deterministic, no LLM, no network (engines are injected — pi-ai's faux provider scripted per test; `globalThis.fetch` stubbed where transport matters): + +1. **Pure core units** — merger (via the real fs adapters, backup-and-restored), derivation, coverage, question grading, plan tree, phases, clusters, layout, tool-parts, prices. +2. **Server flows** — `handleChatRequest` driven end-to-end with scripted engines: persistence, liveness, steer claiming, the wake pipeline (409 retry, quiz deferral, idempotency), crash recovery, the data-dir seam (including a fresh child process). +3. **Sub-agent loops** — researcher/demo-builder/visual-maker with fake headless APIs; the extraction pipeline end-to-end (audit drops, failure-checkpoints-nothing, single-flight, abort). +4. **Render tests** — transcript, cards, docks, composer, sidebar, coverage via React server-side rendering. + +## Lineage + +The teaching methodology comes from an Obsidian-based POC (its `knowledge graph design.md` was the original epistemic-model source, since amended — see the amendments table in `docs/vision.md`). The harness was originally built on the pi agent SDK, pivoted to standalone Next.js + the Vercel AI SDK, and later swapped its LLM engine back to pi-ai while keeping the AI SDK's `UIMessage` as the client protocol ([`docs/pi-engine-swap.md`](docs/pi-engine-swap.md)). + +## Data + +Nothing under `data/` is ever tracked by git. Chats, graph ledgers, jobs, positions, and the settings records are plain files (JSON/JSONL) — inspect them freely; they are the canonical records. The desktop app relocates the whole directory via `LH_DATA_DIR`. \ No newline at end of file