Extraction tool calls: tolerate omitted-empty fields (weak-model trap)

Diagnosis from the real 25-file notes import (user-pasted session logs):
almost every run hit 'Validation failed for tool submit_findings/review:
edges must have required properties edges' — the model (glm flash, low
thinking) omits top-level fields when they're empty, and pi-ai's
structural JSON-Schema validation runs BEFORE the zod layer, so each
omission cost one of the extractor's maxTurns 4 (full payload
regeneration) and occasionally burned out the run entirely (the two
FAILED imports).

The model is technically wrong, but the schema provokes it:
required-but-usually-empty arrays are a known weak-model anti-pattern.

- lhTool's model-facing declaration now converts with io: "input"
  (the declaration describes what the MODEL sends): .default() fields
  drop out of required. No-op for teacher tools (none use defaults).
- findingsSchema.edges / auditSchema.dropped: .default([]) — omission
  means "none"; attribution: .default({quiz:{}}) for quiz-free sessions.
- Prompts hardened in all three agents: include every top-level field,
  never omit because empty.

Structural probe: validateToolArguments now passes the exact pasted
failure shape; zod re-parse in execute applies the defaults. 355/355
tests, typecheck, lint, build green. (lh-tool.ts is the shared engine
seam — flagged to master: io:"input" is the correct mode for every
model-facing declaration, teacher path verified unaffected.)
This commit is contained in:
amosblomqvist
2026-09-13 02:32:20 +03:00
parent 543cf33789
commit c3cc55c585
2 changed files with 25 additions and 6 deletions
+17 -4
View File
@@ -167,7 +167,9 @@ const findingsSchema = z.object({
.describe("Every distinct proposition the session touched — true statements, misconception-free."),
edges: z
.array(edgeSchema)
.describe("Relations between propositions: follows (derivable) / supports (grounds without entailing)."),
.describe("Relations between propositions: follows (derivable) / supports (grounds without entailing). Pass [] when there are none — never omit the field.")
.default([]), // weak models omit required-but-empty arrays; omission means "none"
attribution: z
.object({
quiz: z
@@ -176,7 +178,8 @@ const findingsSchema = z.object({
"toolCallId of EVERY quiz line → the proposition id it was really testing. Attribute RETROACTIVELY — the whole transcript is visible; map each quiz to the proposition it tests (often articulated elsewhere in the session). null when genuinely unattributable.",
),
})
.describe("Quiz attribution — your only event judgment."),
.describe("Quiz attribution — your only event judgment. Pass { quiz: {} } when the session has no quiz lines — never omit the field.")
.default({ quiz: {} }), // weak models omit empty objects too
});
const NOTES_EXTRACTOR_INSTRUCTIONS = `You are the extraction specialist in a learning application. You read ONE source —
@@ -215,6 +218,9 @@ There is no quiz attribution for notes — the source has no quiz events, and th
Evidence discipline: every quote must appear VERBATIM in the source. The auditor rejects anything unbacked. When in
doubt, report; the auditor decides.
Include EVERY top-level field of the call even when empty — "edges": [] when there are no relations. NEVER omit
a field because it is empty.
Your entire deliverable is ONE submit_findings call.`;
const EXTRACTOR_INSTRUCTIONS = `You are the extraction specialist in a learning application. You read ONE session
@@ -259,6 +265,9 @@ quiz outcomes carry that the learner believed the wrong version.
Evidence discipline: every quote must appear VERBATIM in the transcript view (a QUIZ line counts as the
transcript). The auditor rejects anything unbacked. When in doubt, report; the auditor decides.
Include EVERY top-level field of the call even when empty — "edges": [], "attribution": { quiz: {} } when there
are no quizzes. NEVER omit a field because it is empty.
Your entire deliverable is ONE submit_findings call.`;
/** One single-submit agent run — the shared shape of EVERY LLM stage
@@ -377,7 +386,8 @@ const auditSchema = findingsSchema.extend({
reason: z.string(),
}),
)
.describe("Everything you dropped, with the reason — the audit trail."),
.describe("Everything you dropped, with the reason — the audit trail. Pass [] when you dropped nothing — never omit the field.")
.default([]), // same weak-model tolerance as findings.edges
});
// The notes front door's shapes: everything session findings have, minus
@@ -424,7 +434,10 @@ Also drop propositions not referenced by anything else ONLY if they have no othe
well-evidenced proposition is still knowledge.
Never invent new candidates. Your entire deliverable is ONE submit_review call containing the verified findings
(the same shapes you received, with linkTo added) plus the dropped list.`;
(the same shapes you received, with linkTo added) plus the dropped list.
Include EVERY top-level field of the call even when empty — "edges": [], "dropped": []. NEVER omit a field
because it is empty.`;
/** The current graph, compact — the auditor's identity context. */
function graphView(nodes: GraphNode[]): string {