sorokin.oleg.edu/learnharness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Wave A: pi-native tool layer — delete LhTool/toPiTool, TypeBox everywhere, terminate on questions

Browse Source 
- src/agent/lh-tool.ts deleted: the zod-wrapped LhTool + toPiTool bridge
  (z.toJSONSchema per call, clientQuestion flag, hand-rolled result mapping)
  is gone. Tools are pi-agent-core native AgentTools now.
- New src/agent/tool-execute.ts owns what the wrapper added: the abort guard
  (runWithAbortGuard: run signal + 30s per-call timeout + 250ms grace,
  'Operation aborted' / timeout wording preserved), ToolInputError (the
  retryable '- problem at path' format), result mapping (string verbatim /
  object JSON / image parts, details rides the record's tool part output),
  and questionResult() — placeholder + terminate:true (owner Option A:
  belt of terminate + slim makeQuestionStopper with the clean-question
  rule; maxTurns deleted from the teaching turn entirely per owner).
- All 11 tool files + tools.ts registries converted to TypeBox parameters
  (typebox@1.3.7, pinned to pi-agent-core's copy). zod refinements
  (quizInputProblems, askOptionProblems, live-nodeId) moved from schemas
  into execute — the model sees the same retryable error shape.
- question-core + contracts/probe TypeBox-authored (single source shared
  with the UI); tool-parts parse seam runs the boundary's full chain
  (Convert → Check → Clean → refinement), fixing the live-smoke lockout:
  pi's boundary Value.Converts correctAnswer: 17 → '17' but the record
  keeps raw args, so a parse without Convert rejected a posed quiz
  (invisible pending question, turn ended). Pinned with the real smoke
  emission shape.
- Subagents converted: demo-builder/researcher AgentTool-native,
  visual-maker/pdf-reader declaration-only pi-ai Tools with local execute
  fns, extraction submit schemas re-authored in TypeBox (owner option 2:
  zod defaults → Type.Optional + fill at the submit seam).
- Accepted delta, flagged: TypeBox-native schemas skip pi's JSON-Schema
  primitive coercion fast path... (boundary still Value.Converts), and
  prober-quiz unknown keys ride through (Value.Clean strips at display).
- Tests: 988 passing (was 967 at the recon baseline); tool-abort and
  read-tool rewritten against the native seams; question-core/question-
  tools/tool-parts/demos/web-search/graph-tools/plan-tree/probe-done/
  tool-toggles converted to Value.Check/execute pins.

Live smoke (PORT=3004, real model): native ls turn streams end-to-end; a
real quiz pose (first attempt missing correctAnswer → tool-input-error →
model retry → posed) terminates the turn cleanly and persists pending.
This commit is contained in:
amosblomqvist
2026-09-25 23:32:13 +03:00
parent b07cb313f0
commit b87a6925af
35 changed files with 1357 additions and 1083 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/core/contracts/probe.ts
+34 -32
View File
@@ -15,7 +15,7 @@
* the learner's edge sits.
*/
import { z } from "zod";
import { Type, type Static } from "typebox";
/** One dependency strand the planned lesson will rest on. */
export interface ProbeStrand {
@@ -38,41 +38,43 @@ export interface ProbeDoneInput {
assessment: ProbeAssessment;
}
// The zod schema of that input — SINGLE SOURCE OF TRUTH, lived by the tool
// (src/agent/tools/probe.ts) and by the UI parse seam (src/core/tool-parts.ts):
// The TypeBox schema of that input — SINGLE SOURCE OF TRUTH, lived by the
// tool (src/agent/tools/probe.ts) and by the UI parse seam (src/core/tool-parts.ts):
// the boundary validation and the consumer parse run literally the same
// schema, so "schema-valid at the boundary" ⇔ "parseable by a consumer" is
// structural, not a convention. (Live crash 2026-09-12: the model sent
// `assessment` as a STRING; the card cast it and died on `.strands.map`.)
export const probeDoneInputSchema = z
.object({
assessment: z
.object({
strands: z
.array(
z.object({
strand: z
.string()
.describe(
"The dependency strand (e.g. 'the derivative as a limit') — a name the plan can hang a node on.",
),
assessment: z
.string()
.describe(
"Where his edge sits on this strand: the floor (what he reliably gets right), the ceiling (where it runs out), " +
"misconceptions found, and what the lesson should build from.",
),
}),
)
.min(1)
.describe("Every strand the planned lesson will depend on, each with your read of his edge on it."),
summary: z
.string()
.optional()
.describe("Optional overall synthesis: his apparent goal and where the teaching should enter."),
})
.describe("The learner assessment — the probe's distilled output."),
}) satisfies z.ZodType<ProbeDoneInput>;
export const probeDoneInputSchema = Type.Object({
assessment: Type.Object(
{
strands: Type.Array(
Type.Object({
strand: Type.String({
description:
"The dependency strand (e.g. 'the derivative as a limit') — a name the plan can hang a node on.",
}),
assessment: Type.String({
description:
"Where his edge sits on this strand: the floor (what he reliably gets right), the ceiling (where it runs out), " +
"misconceptions found, and what the lesson should build from.",
}),
}),
{ minItems: 1, description: "Every strand the planned lesson will depend on, each with your read of his edge on it." },
),
summary: Type.Optional(
Type.String({
description: "Optional overall synthesis: his apparent goal and where the teaching should enter.",
}),
),
},
{ description: "The learner assessment — the probe's distilled output." },
),
});
/** Schema ↔ interface lock (the old `satisfies z.ZodType` guarantee): the
* declaration and ProbeDoneInput can never drift apart — this only compiles
* when the schema's static shape IS the interface. */
export const probeDoneSchemaLock = (v: Static<typeof probeDoneInputSchema>): ProbeDoneInput => v;
/** probe_done tool output. */
export interface ProbeDoneResult {
src/core/question-core.ts
+110 -107
View File
@@ -2,10 +2,10 @@
* question-core — the question contract + pure grading logic. No framework
* imports, no UI: everything here is deterministic and unit-testable.
*
* The zod schemas below are the tool INPUT contracts (agent/tools declares
* the tools against them); the browser validates, shuffles, and grades with
* these functions, then sends the verdict back as the tool output via
* addToolOutput:
* The TypeBox schemas below are the tool INPUT contracts (agent/tools
* declares the tools against them); the browser validates, shuffles, and
* grades with these functions, then sends the verdict back as the tool
* output via addToolOutput:
* 1. The teacher model calls the `quiz` tool (question + options +
* correctAnswer + explanation). No server execute → the call lands in
* the UI as a typed tool part in `input-available` state.
@@ -17,7 +17,7 @@
* exited and resumed mid-question for free.
*/
import { z } from "zod";
import { Type, type Static } from "typebox";
// ─── shared types ───────────────────────────────────────────────────────────
@@ -30,14 +30,18 @@ export interface QuizOption {
// ─── tool input schemas (the question contract) ─────────────────────────────
// Parameter names mirror the prompt docs exactly — keep them in sync.
const optionSchema = z.object({
label: z.string().describe("Display label for the answer option"),
value: z
.string()
.optional()
.describe("Optional machine-readable value returned for the option. Defaults to the label."),
description: z.string().optional().describe("Optional extra detail shown below the option"),
});
/** One answer option — shared by quiz + ask. */
const optionProps = {
label: Type.String({ description: "Display label for the answer option" }),
value: Type.Optional(
Type.String({
description: "Optional machine-readable value returned for the option. Defaults to the label.",
}),
),
description: Type.Optional(
Type.String({ description: "Optional extra detail shown below the option" }),
),
};
// ─── cross-field authoring validation (tool-schema refinements) ──────────
// Enforced at the tool-call boundary (see the superRefines below): an invalid
@@ -111,114 +115,113 @@ export function quizInputProblems(input: {
// and teach agents use DIFFERENT tools under the same name (docs/probe-teach-
// split.md §1): the prober's quiz has NO nodeId field at all (there is no
// map to position on), the teacher's adds it (the position signal — the
// per-session tree validation chains on top in tools.ts). Grading rules
// are identical, single-sourced here.
export const quizInputBase = z.object({
question: z.string().describe("The single quiz question to ask. ONE question per turn: ask, then end your turn — the answer determines the next question."),
options: z
.array(optionSchema)
.min(2)
.describe(
// per-session tree validation runs in the tool's execute, tools.ts).
// Grading rules are identical, single-sourced here. The cross-field
// authoring contract (quizInputProblems, above) is enforced at the
// tool-call boundary in execute — TypeBox carries the structure to the
// model, the problems function carries the refinements.
//
// Argument-order ruling (owner, 2026-09-25): the explanation is REASONING,
// and reasoning happens BEFORE the answer is chosen — models emit tool
// args in schema order, so explanation sits immediately BEFORE correctAnswer.
// (All consumers read by key; the order is authoring guidance, nothing else.)
export const quizInputBase = Type.Object({
question: Type.String({
description:
"The single quiz question to ask. ONE question per turn: ask, then end your turn — the answer determines the next question.",
}),
options: Type.Array(Type.Object(optionProps), {
minItems: 2,
description:
"The answer options (2 or more). Options only — there is no free-text mode. " +
"Give each option a stable `value`; you reference the correct one by that value in correctAnswer. " +
"Never add your own 'I don't know' option — that choice is added for you.",
),
multiSelect: z
.boolean()
.optional()
.describe("Set to true when more than one option is correct and the learner must select all of them."),
// Argument-order ruling (owner, 2026-09-25): the explanation is REASONING,
// and reasoning happens BEFORE the answer is chosen — models emit tool
// args in schema order, so explanation sits immediately BEFORE correctAnswer.
// (All consumers read by key; the order is authoring guidance, nothing else.)
explanation: z
.string()
.describe(
"Give each option a stable `value`; you reference the correct one by that value in correctAnswer. " +
"Never add your own 'I don't know' option — that choice is added for you.",
}),
multiSelect: Type.Optional(
Type.Boolean({
description: "Set to true when more than one option is correct and the learner must select all of them.",
}),
),
explanation: Type.String({
description:
"REQUIRED. Explanation revealed AFTER the learner answers (shown whether they got it right or wrong). " +
"Use it to reinforce why the correct answer is correct. It is written BEFORE correctAnswer by design: " +
"reason it out first, then commit the answer the reasoning supports.",
),
correctAnswer: z
.union([z.string(), z.array(z.string())])
.describe(
"Use it to reinforce why the correct answer is correct. It is written BEFORE correctAnswer by design: " +
"reason it out first, then commit the answer the reasoning supports.",
}),
correctAnswer: Type.Union([Type.String(), Type.Array(Type.String())], {
description:
"REQUIRED. The correct answer as the option value(s) — the `value` field of the option you intend. " +
'Single-select: a single string (e.g. "mercury"). Multi-select: an array of strings (e.g. ["belize", "niue"]); ' +
"the learner is only correct if their selection matches this set exactly. " +
"Always pass the value, not a position number — this is self-checking and prevents miscounting. " +
"A value that matches no option fails validation and the call is returned to you to fix.",
),
details: z.string().optional().describe("Optional extra context or instructions shown under the question."),
'Single-select: a single string (e.g. "mercury"). Multi-select: an array of strings (e.g. ["belize", "niue"]); ' +
"the learner is only correct if their selection matches this set exactly. " +
"Always pass the value, not a position number — this is self-checking and prevents miscounting. " +
"A value that matches no option fails validation and the call is returned to you to fix.",
}),
details: Type.Optional(
Type.String({ description: "Optional extra context or instructions shown under the question." }),
),
});
// The TEACH-side quiz object: the shared base + the plan-position field.
// `nodeId` means "this question is ABOUT this node" — a position signal,
// never a completion signal (docs/probe-teach-split.md §3).
export const quizInputObject = quizInputBase.extend({
nodeId: z
.string()
.optional()
.describe(
"Optional plan-tree node id this question is about — must EXACTLY match one of this session's " +
// never a completion signal (docs/probe-teach-split.md §3). The cross-field
// authoring contract (the Gate-3 hardening: a quiz whose correctAnswer
// can't be graded NEVER reaches the learner — schema-valid ⇔ gradeable, the
// rules exactly what the grading path accepts) runs at the CALL BOUNDARY in
// the tool's execute: quizInputProblems here, the per-session plan-nodeId
// validation in tools.ts (it reads the live chat record, which this core
// never touches).
export const quizInputObject = Type.Object({
...quizInputBase.properties,
nodeId: Type.Optional(
Type.String({
description:
"Optional plan-tree node id this question is about — must EXACTLY match one of this session's " +
"plan_init node slugs (validated at call time; a mismatched id is rejected back to you). " +
"Tag every question that targets a map node: the lesson's position on the map derives from these tags " +
"(the node the latest tagged question is about is where the map stands; earlier ones count as visited). " +
"Omit for quick ad-hoc checks that don't target a node.",
),
})
// The cross-field authoring contract (the Gate-3 hardening): a quiz whose
// correctAnswer can't be graded NEVER reaches the learner. The SDK validates
// client-side tool inputs against this schema at the call boundary and, on
// failure, returns the validation error to the teacher as a tool result —
// the teacher retries with a fixed quiz in the same turn. Schema-valid ⇔
// gradeable: the rules are exactly what the grading path accepts.
//
// NOTE the split: the raw objects (pre-refinement) are exported for
// composition — tools.ts teachQuestionTools chains the per-session
// plan-nodeId validation (it reads the live chat record, which this core
// never touches) on top of this shared refinement, and the PROBE quiz
// (no nodeId at all) refines quizInputBase directly.
export const quizInputSchema = quizInputObject
.superRefine((input, ctx) => {
for (const problem of quizInputProblems(input)) {
ctx.addIssue({ code: "custom", message: problem });
}
});
/** The PROBE-side quiz schema: the shared authoring contract, no nodeId
* field at all — the session's map doesn't exist yet. */
export const probeQuizInputSchema = quizInputBase.superRefine((input, ctx) => {
for (const problem of quizInputProblems(input)) {
ctx.addIssue({ code: "custom", message: problem });
}
}),
),
});
export const askInputSchema = z
.object({
// No `mode` param — the pi original has none: the mode is inferred from
// whether options are present (the dock does exactly this). One less
// knob for the model to misuse.
question: z.string().describe("The single question to ask the learner. ONE question per turn: ask, then end your turn — the answer determines the next question."),
details: z.string().optional().describe("Optional extra context or instructions shown under the question."),
options: z
.array(optionSchema)
.min(2)
.optional()
.describe(
"Optional multiple-choice options. Omit for free-form text input. " +
"The learner can always choose 'Other' and type a custom answer when options are provided.",
),
multiSelect: z.boolean().optional().describe("Set to true to allow multiple answers to the same question."),
})
// Same option sanity as the quiz (labels non-empty, values distinct — ask
// is not graded, but duplicate values break the dock's value keying).
.superRefine((input, ctx) => {
for (const problem of askOptionProblems(input.options ?? [])) {
ctx.addIssue({ code: "custom", message: problem });
}
});
/** The PROBE-side quiz declaration: the shared authoring contract, no
* nodeId field at all — the session's map doesn't exist yet. Structurally
* identical to quizInputBase (the old superRefine never reached the wire —
* it ran at the call boundary, where quizInputProblems runs now). */
export const probeQuizInputSchema = quizInputBase;
export type QuizToolInput = z.infer<typeof quizInputSchema>;
export type AskToolInput = z.infer<typeof askInputSchema>;
/** The TEACHER's quiz declaration (base + the position field). */
export const quizInputSchema = quizInputObject;
export const askInputSchema = Type.Object({
// No `mode` param — the pi original has none: the mode is inferred from
// whether options are present (the dock does exactly this). One less
// knob for the model to misuse.
question: Type.String({
description:
"The single question to ask the learner. ONE question per turn: ask, then end your turn — the answer determines the next question.",
}),
details: Type.Optional(
Type.String({ description: "Optional extra context or instructions shown under the question." }),
),
options: Type.Optional(
Type.Array(Type.Object(optionProps), {
minItems: 2,
description:
"Optional multiple-choice options. Omit for free-form text input. " +
"The learner can always choose 'Other' and type a custom answer when options are provided.",
}),
),
multiSelect: Type.Optional(
Type.Boolean({ description: "Set to true to allow multiple answers to the same question." }),
),
});
// Ask's option sanity (labels non-empty, values distinct — ask is not
// graded, but duplicate values break the dock's value keying) runs at the
// call boundary in execute (askOptionProblems, above).
export type QuizToolInput = Static<typeof quizInputObject>;
export type AskToolInput = Static<typeof askInputSchema>;
/** A real selected answer, in the order the options were displayed. */
src/core/tool-parts.ts
+49 -26
View File
@@ -3,9 +3,9 @@
* live crashes).
*
* WHY THIS EXISTS. Validation IS first at the model-facing boundary — pi
* validates arguments before execute, and our zod re-run adds the
* refinements; a failure returns to the model as an error tool result,
* same turn. But the same bytes have two other consumers, and there they
* validates arguments against the tool's TypeBox schema before execute,
* and the authoring refinements run in the execute seam; a failure
* returns to the model as an error tool result, same turn. But the same bytes have two other consumers, and there they
* exist UNVALIDATED BY DESIGN:
*
* - the live wire streams raw tool-input deltas while the model is still
@@ -20,49 +20,72 @@
* lie the checker believes, with no runtime check behind it.
*
* THE RULE (mechanical, not vigilance): model-authored INPUT is consumed
* only through these parsers, each running the SAME zod schema the model
* only through these parsers, each running the SAME TypeBox schema the model
* faced — boundary-valid ⇔ consumer-parseable, structurally. Server- and
* learner-authored OUTPUT stays trusted (our own code authors it: the
* execute acks, the learner's buildAnswer). A parse failure renders the
* quiet fallback, never a crash.
*/
import { z } from "zod";
import { quizInputBase, askInputSchema, type AskToolInput, type QuizToolInput } from "./question-core.ts";
import { Value } from "typebox/value";
import type { TSchema } from "typebox";
import { quizInputBase, askInputSchema, askOptionProblems, quizInputProblems, type AskToolInput, type QuizToolInput } from "./question-core.ts";
import { probeDoneInputSchema, type ProbeDoneInput } from "./contracts/probe.ts";
/** The pi-boundary parse step, mirrored here (live-smoke fix, 2026-09-26):
* pi-ai's validateToolArguments runs TypeBox's Value.Convert on the model's
* raw arguments before Value.Check — a correctAnswer sent as the NUMBER 17
* (what the live smoke's model actually emitted) converts to "17" at the
* boundary, so the tool poses the quiz fine. But the persisted record keeps
* the RAW arguments, so a parse seam that skips Convert would reject the
* exact input the boundary accepted — an invisible pending quiz and a
* learner lockout. Every display parse converts first, exactly like the
* boundary; Convert mutates, so the input is cloned before touching. */
function boundaryParse<T>(schema: TSchema, input: unknown): T | undefined {
if (typeof input !== "object" || input === null) return undefined;
const converted = structuredClone(input);
Value.Convert(schema, converted);
if (!Value.Check(schema, converted)) return undefined;
return Value.Clean(schema, converted) as T;
}
/**
* Quiz parse for DISPLAY: the shared authoring base, pre-refinement.
* Both boundary schemas (the prober's probeQuizInputSchema over the bare
* base, the teacher's quizInputSchema over base + optional nodeId) refine
* THE SAME quizInputBase, so anything that passed either boundary parses
* here: refinements don't weaken on re-run, and the teacher-only `nodeId`
* is an unknown key the base strips. This is deliberately the LENIENT
* common core — the strict per-agent refinements are the boundary's job.
* Both boundaries (the prober's quizInputBase, the teacher's quizInputObject
* over base + optional nodeId) share THE SAME quizInputBase fields, so
* anything that passed either boundary parses here; the teacher-only
* `nodeId` is an unknown key Value.Clean strips (zod's old strip behavior,
* kept so the display shape stays exactly what the dock always saw). This
* is deliberately the LENIENT common core — the strict per-agent authoring
* checks are the boundary's job — but it runs the shared grading
* refinement (quizInputProblems) after Convert, mirroring the boundary's
* full chain: pi accepts `correctAnswer: 17` structurally (Convert →
* "17") and execute's refinement is what makes it a posed-or-rejected
* call. The display parse follows the same order so an input that could
* never have been posed (e.g. `correctAnswer: undefined`, which Convert
* widens to "") renders as a quiet fallback, not a broken quiz.
*/
export function parseQuizInput(input: unknown): QuizToolInput | undefined {
const parsed = quizInputBase.safeParse(input);
return parsed.success ? (parsed.data as QuizToolInput) : undefined;
const cleaned = boundaryParse<QuizToolInput>(quizInputBase, input);
if (!cleaned) return undefined;
if (quizInputProblems(cleaned).length > 0) return undefined;
return cleaned;
}
/**
* Ask parse for DISPLAY: the full ask schema (options optional — free-text
* mode is the options-less shape). The only refinement is option sanity
* (distinct values), which the boundary already enforced.
* mode is the options-less shape), plus the option-sanity refinement the
* boundary enforced (askOptionProblems — duplicate values break the dock's
* keying, so they parse to undefined exactly like the zod seam did).
*/
export function parseAskInput(input: unknown): AskToolInput | undefined {
const parsed = askInputSchema.safeParse(input);
return parsed.success ? parsed.data : undefined;
const cleaned = boundaryParse<AskToolInput>(askInputSchema, input);
if (!cleaned) return undefined;
if (askOptionProblems(cleaned.options ?? []).length > 0) return undefined;
return cleaned;
}
/** probe_done parse — the core-owned schema the tool itself declares. */
export function parseProbeDoneInput(input: unknown): ProbeDoneInput | undefined {
const parsed = probeDoneInputSchema.safeParse(input);
return parsed.success ? parsed.data : undefined;
}
/** Parse helper for schemas this seam doesn't own (kept for future cards). */
export function parseWith<T>(schema: z.ZodType<T>, input: unknown): T | undefined {
const parsed = schema.safeParse(input);
return parsed.success ? parsed.data : undefined;
return boundaryParse<ProbeDoneInput>(probeDoneInputSchema, input);
}