feat(models): BYOK all the way down — factory pins deleted, per-need refusal gates

Owner decision 2026-09-22: 'it doesn't make sense to have any default
model configured anywhere; the application is bring your own keys.'

config.ts: MODELS deleted; ModelRole becomes an explicit 9-role union;
DEFAULT_THINKING_ORDER (medium, low, high, minimal, xhigh, max) bakes
the unset-level default — first entry the model supports; explicitly
set levels keep pi's native nearest-clamp.

core/settings-resolution: the factory layer dissolves — the chain is
role override → slot → NOTHING. resolveSettings(record) takes no
factory. Empty chains resolve to null/undefined; a thinkingLevel-only
slot is inert (stored, never run). resolveConversationSpec: chat.model
override over the strong slot, null when both empty. resolveRoleSpec
spans all nine roles for the job/extraction engines.

Refusal envelope (core/settings.ts): modelsNotConfigured(missing) —
{error, code: 'models_not_configured', missing: [slot tiers]} +
ModelsNotConfiguredError with .envelope. Per-need gates, each honest:
- chat-handler: universal turn gate (new + continuing turns) returns
  503 before beginTurn — no liveness, no write-ahead persist, no graph
  trigger before the gate fires.
- runner.runJob: unresolvable engine lands the job as error with the
  envelope wording; no phantom loop, no result steer.
- extraction: refuseIfStagesUnconfigured throws before any stage
  (turn-service info-skips so coverage retries next turn; the import
  route surfaces per-file code+missing); labeler-less runs skip
  cluster naming with a console line and still merge.

Write side: slot entries must carry a model — level-only slot writes
are 400s (absent slot key = clear). piEngineForSpec resolves unset
thinking levels via DEFAULT_THINKING_ORDER; piEngineFor now throws
honestly; thinkingLevelFor/modelIdFor deleted; teacher-agent, pi-run,
visual-maker, pdf-reader all read engine.level.

model-catalog: defaults/slots/rolePins null when the chain is empty
(contracts nullable) — no phantom ids in payloads. Stores drop
MODEL_PINS entirely.

Tests: pin-leaners converted to explicit injection (per-chat model
pins on scripted chats — exactly what the setup flow writes; faux
engines for agent-construction tests). New coverage: the 503 gate
envelope, the runner refusal, the extraction refusal + labeler skip.

Gates: typecheck, lint, 815/815, build.
This commit is contained in:
amosblomqvist
2026-09-22 16:59:16 +03:00
parent 8675b19bef
commit bfa906e2f8
26 changed files with 889 additions and 562 deletions
+42
View File
@@ -41,6 +41,7 @@ import type {
} from "../src/core/contracts.ts";
import type { GraphStore } from "../src/core/ports.ts";
import { abortExtraction, createExtractionPipeline, extractionStatuses, transcriptView } from "../src/agent/subagents/extraction.ts";
import { ModelsNotConfiguredError } from "../src/core/settings.ts";
import { batchTag, coverageOf, coverageSlots } from "../src/core/coverage.ts";
import { clusterNamingSettled, refreshClusterLabels } from "../src/agent/subagents/cluster-namer.ts";
import { semanticClusters } from "../src/core/semantic-clusters.ts";
@@ -394,6 +395,7 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
assert.equal(covered.length, record.messages.length);
assert.ok(covered.every((c) => c.chatId === chatId && c.fp.length === 64 && c.tokens > 0));
assert.equal(ledgers.readCheckpoints().length, 0, "the chat path writes coverage, not the legacy checkpoint");
assert.equal(
coverageOf(record.messages as UIMessage[], covered, chatId).uncoveredCount,
0,
@@ -401,6 +403,46 @@ describe("the extraction pipeline (extractor → auditor → merger)", () => {
);
});
it("BYOK refusal: no stage engines anywhere → the run throws honestly before any stage (owner, 2026-09-22)", async () => {
const { chatId } = fixtureChat();
const ledgers = memoryLedgers();
// No direct engines AND no engineFor — the chain came up empty.
const pipeline = createExtractionPipeline({ chatStore, graphStore: ledgers, embedder: fakeEmbedder });
await assert.rejects(pipeline.runExtraction(chatId), (err: unknown) => {
assert.ok(err instanceof ModelsNotConfiguredError);
assert.deepEqual(err.envelope.missing, ["cheap"]); // extractor + auditor are the cheap tier
assert.equal(err.envelope.code, "models_not_configured");
return true;
});
// nothing ran: no phase residue, no usage, no coverage writes
assert.equal(ledgers.readCoverage().length, 0);
});
it("a labeler-less run merges fine and SKIPS cluster naming honestly (BYOK, 2026-09-22)", async () => {
const { chatId } = fixtureChat();
const ledgers = memoryLedgers();
// Identity-pass verdict: p1 becomes a new node, p2 dropped (no links).
const verdict: AuditResult = {
propositions: [{ ...FINDINGS.propositions[0]! }],
edges: [],
attribution: { quiz: { "probe-1": "p1" } },
dropped: [{ ref: "p2", reason: "evidence not found" }],
};
const pipeline = createExtractionPipeline({
chatStore,
graphStore: ledgers,
embedder: fakeEmbedder,
extractor: submitModel("submit_findings", FINDINGS),
auditor: submitModel("submit_review", verdict),
// NO labeler: the cluster-naming garnish skips (console line),
// the merge itself is unaffected.
});
const stats = await pipeline.runExtraction(chatId);
assert.deepEqual(stats, { nodes: 1, edges: 0, events: 3 });
});
it("engineFor fills the stages (model slots wave): the chain replaces config pins", async () => {
const { chatId } = fixtureChat();
const ledgers = memoryLedgers();