fix: quiet Retrying lines, quiz option descriptions nuked, dividers at the markers

Three owner rulings from the gradient-descent session
(chat_muhskq6p_e8c3600b — 7 failed quiz calls, answers leaked in
option descriptions, planning divider floating at the transcript top):

1. RETRYING LINES (general): any tool call that errors back to the
   agent (validation failure, malformed args — the retry loop) renders
   one quiet "Retrying…" SystemLine instead of silence. Question tools
   previously rendered null, so a flailing quiz loop read as a stall.
   Aborts are terminal ("Operation aborted") and stay silent. Verified
   live: exactly 7 lines render at the 7 failure sites of the session.

2. QUIZ OPTION DESCRIPTIONS: NUKED. The model used them to telegraph
   correctness in every quiz of the session ("that moves uphill — it
   climbs f instead of descending it") — answerable by elimination.
   The field is gone from BOTH quiz schemas (probe + teach); ask keeps
   it (ungraded — nothing to leak). The dock additionally suppresses
   descriptions for quiz kind, so pre-fix records stop leaking too.
   Verified live: the session's pending quiz renders bare labels.

   Plus the diagnosed failure-mode fixes: GLM sometimes emits `options`
   as a JSON-encoded string (6 of 7 failures). prepareArguments hooks
   on all three question tools coerce it BEFORE pi's validation (the
   call lands instead of bouncing); the client coerces again in
   normalizeOptions (pi persists raw call args). Schema descriptions
   hardened ("REQUIRED ... never a JSON-encoded string").

3. DIVIDERS AT THE MARKERS: the probe-handoff SystemLine
   ("Preparing your plan…") IS NOW the Planning divider; the accepted
   verdict IS NOW the Teaching divider ("✓ Plan accepted" line gone) —
   one signal per seam, not two. The fold-based divider block is
   DELETED: Wave D grouping merged the whole probe into one giant
   assistant group, so the inclusive-fold comparison floated the
   divider to the transcript top (reproduced against the real record:
   6 view messages, phase flipped at the giant group). The markers
   render in the right spot by record construction. A DECLINED verdict
   is not a seam — it keeps the "↻ Plan revision requested" line.
   Verified live: plan divider after the probe concluded, teach
   divider after the plan presentation, nothing at the top.

Tests: 1004/1004 — string-coercion pins (normalizeOptions + the
prepareArguments hooks on all three tools, identity fast path),
quiz-schema-has-no-description pin (ask keeps it).
This commit is contained in:
amosblomqvist
2026-09-26 19:08:39 +03:00
parent b84ab1cdc0
commit 9853075331
6 changed files with 186 additions and 44 deletions
+29
View File
@@ -8,6 +8,7 @@ import { Value } from "typebox/value";
import type { TSchema } from "typebox";
import {
askInputSchema,
quizInputBase,
askOptionProblems,
buildAnswer,
coerceCorrectAnswer,
@@ -184,6 +185,34 @@ describe("normalizeOptions", () => {
it("throws on duplicate option values", () => {
assert.throws(() => normalizeOptions([{ label: "a", value: "x" }, { label: "b", value: "x" }]), /duplicate/);
// GLM wobble guard (2026-09-26 gradient-descent session): the model
// sometimes emits `options` as a JSON-encoded string — 6 of 7 quiz
// validation failures in that session. The dock (and grading) coerce
// via the same parseOptionsIfString the server's prepareArguments
// runs pre-validation.
const asString = JSON.stringify([{ label: "Slope", value: "slope" }, { label: "Value", value: "value" }]);
const coerced = normalizeOptions(asString);
assert.deepEqual(coerced, [
{ label: "Slope", value: "slope", description: undefined },
{ label: "Value", value: "value", description: undefined },
], "a JSON-encoded string coerces to the same normalized options");
// Non-JSON strings pass through and behave like no options (the
// server's schema validation reports the honest error instead).
assert.deepEqual(normalizeOptions("not json"), []);
});
it("quiz options carry NO description field; ask keeps it (owner nuke, 2026-09-26)", () => {
// The model used quiz option descriptions to telegraph correctness
// ("that's not the gradient") in every quiz of the gradient-descent
// session — the field is gone from BOTH quiz schemas; ask (ungraded,
// nothing to leak) keeps it.
const quizOption = (quizInputBase.properties.options as { items?: { properties?: Record<string, unknown> } })
.items!.properties!;
assert.ok(!("description" in quizOption), "quiz option schema has no description");
const askOption = (askInputSchema.properties.options as { items?: { properties?: Record<string, unknown> } })
.items!.properties!;
assert.ok("description" in askOption, "ask option schema keeps description");
});
it("handles undefined input", () => {
+44 -1
View File
@@ -15,7 +15,14 @@ import * as fs from "node:fs";
import * as path from "node:path";
import { after, describe, it } from "node:test";
import { Value } from "typebox/value";
import { createProberTools, createTeacherTools, teachQuizToolFor, type TeacherToolContext } from "../src/agent/tools.ts";
import {
askUserQuestionTool,
createProberTools,
createTeacherTools,
probeQuizTool,
teachQuizToolFor,
type TeacherToolContext,
} from "../src/agent/tools.ts";
import { TOGGLEABLE_TOOLS } from "../src/core/pending.ts";
import { jobStore } from "../src/store/jobs.ts";
import { probeQuizInputSchema } from "../src/core/question-core.ts";
@@ -231,3 +238,39 @@ describe("tool toggles (core/pending.ts + factories)", () => {
);
});
});
// ─── prepareArguments: the GLM string-encoded options guard (2026-09-26) ────
describe("prepareQuestionArguments (the string-encoded options wobble)", () => {
// The gradient-descent session (chat_muhskq6p): 6 of 7 quiz validation
// failures were `options` arriving as a JSON-encoded STRING. pi runs
// prepareArguments BEFORE schema validation — the coerced call lands
// instead of bouncing back as a wasted retry turn.
const goodOptions = [
{ label: "The slope of the tangent line", value: "slope" },
{ label: "The value of the curve", value: "value" },
];
const asString = JSON.stringify(goodOptions);
it("coerces string-encoded options on ALL THREE question tools", () => {
for (const tool of [askUserQuestionTool, probeQuizTool, teachQuizToolFor({ chatId: "chat_x", chatStore: undefined as never })]) {
const prepare = (tool as unknown as { prepareArguments?: (a: unknown) => unknown }).prepareArguments;
assert.ok(prepare, `${tool.name} has the hook`);
const out = prepare({ question: "q", options: asString, correctAnswer: "slope", explanation: "e" });
assert.deepEqual(
(out as { options: unknown }).options,
goodOptions,
`${tool.name} coerces the string to an array`,
);
}
});
it("passes everything else through UNTOUCHED (identity — pi's fast path)", () => {
const prepare = (probeQuizTool as unknown as { prepareArguments: (a: unknown) => unknown }).prepareArguments;
const arrayArgs = { question: "q", options: goodOptions, correctAnswer: "slope", explanation: "e" };
assert.equal(prepare(arrayArgs), arrayArgs, "array args return the SAME object (no defensive copies)");
const unparseable = { question: "q", options: "not json", correctAnswer: "slope", explanation: "e" };
assert.equal(prepare(unparseable), unparseable, "unparseable strings pass through — validation reports honestly");
});
});