sorokin.oleg.edu/learnharness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

feat(models): BYOK all the way down — factory pins deleted, per-need refusal gates

Browse Source 
Owner decision 2026-09-22: 'it doesn't make sense to have any default
model configured anywhere; the application is bring your own keys.'

config.ts: MODELS deleted; ModelRole becomes an explicit 9-role union;
DEFAULT_THINKING_ORDER (medium, low, high, minimal, xhigh, max) bakes
the unset-level default — first entry the model supports; explicitly
set levels keep pi's native nearest-clamp.

core/settings-resolution: the factory layer dissolves — the chain is
role override → slot → NOTHING. resolveSettings(record) takes no
factory. Empty chains resolve to null/undefined; a thinkingLevel-only
slot is inert (stored, never run). resolveConversationSpec: chat.model
override over the strong slot, null when both empty. resolveRoleSpec
spans all nine roles for the job/extraction engines.

Refusal envelope (core/settings.ts): modelsNotConfigured(missing) —
{error, code: 'models_not_configured', missing: [slot tiers]} +
ModelsNotConfiguredError with .envelope. Per-need gates, each honest:
- chat-handler: universal turn gate (new + continuing turns) returns
  503 before beginTurn — no liveness, no write-ahead persist, no graph
  trigger before the gate fires.
- runner.runJob: unresolvable engine lands the job as error with the
  envelope wording; no phantom loop, no result steer.
- extraction: refuseIfStagesUnconfigured throws before any stage
  (turn-service info-skips so coverage retries next turn; the import
  route surfaces per-file code+missing); labeler-less runs skip
  cluster naming with a console line and still merge.

Write side: slot entries must carry a model — level-only slot writes
are 400s (absent slot key = clear). piEngineForSpec resolves unset
thinking levels via DEFAULT_THINKING_ORDER; piEngineFor now throws
honestly; thinkingLevelFor/modelIdFor deleted; teacher-agent, pi-run,
visual-maker, pdf-reader all read engine.level.

model-catalog: defaults/slots/rolePins null when the chain is empty
(contracts nullable) — no phantom ids in payloads. Stores drop
MODEL_PINS entirely.

Tests: pin-leaners converted to explicit injection (per-chat model
pins on scripted chats — exactly what the setup flow writes; faux
engines for agent-construction tests). New coverage: the 503 gate
envelope, the runner refusal, the extraction refusal + labeler skip.

Gates: typecheck, lint, 815/815, build.
This commit is contained in:
amosblomqvist
2026-09-22 16:59:16 +03:00
parent 8675b19bef
commit bfa906e2f8
26 changed files with 889 additions and 562 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/core/contracts/models.ts
+26 -18
View File
@@ -26,38 +26,46 @@ export interface ModelCatalogEntry {
maxTokens: number;
}
/** The conversation agents' config pin (config.ts MODELS prober/teacher —
* one controller, so one pin): what a chat without settings runs. */
/** The conversation agents' resolved default (BYOK, owner 2026-09-22): the
* chat's model override over the STRONG slot — one controller, so one
* default. thinkingLevel is optional: unset resolves from the baked
* DEFAULT_THINKING_ORDER at engine build. */
export interface CatalogDefaults {
model: { provider: string; id: string };
thinkingLevel: ThinkingLevel;
thinkingLevel?: ThinkingLevel;
}
/** One background role's RESOLVED pin (settings interlude, 2026-09-19):
* the user's agent-models override falling back to the config.ts MODELS
* pin — what that role actually runs today. The settings UI shows each
* agent-role row's "Default: <this>" caption from it. */
/** One background role's RESOLVED pin (BYOK, owner 2026-09-22): what that
* role actually runs today — role override → tier slot → NOTHING. NULL
* = the role's chain is empty (the per-need gates refuse its work); the
* settings UI shows "not configured" from it. */
export type RolePins = Record<
BackgroundModelRole,
{ model: { provider: string; id: string }; thinkingLevel?: ThinkingLevel }
{ model: { provider: string; id: string }; thinkingLevel?: ThinkingLevel } | null
>;
/** One slot tier's RESOLVED pin (model slots wave, 2026-09-20): the
* strong/cheap/vision tier's concrete model — slot override falling back
* to the factory rep pin. The settings UI captions each slot row with it
* and the picker pre-fills from it. */
export type SlotPins = Record<ModelSlotName, { model: { provider: string; id: string }; thinkingLevel?: ThinkingLevel }>;
/** One slot tier's RESOLVED pin (BYOK, owner 2026-09-22): NULL = the tier
* holds no model. The settings UI captions each slot row with it and the
* picker pre-fills from it. */
export type SlotPins = Record<
ModelSlotName,
{ model: { provider: string; id: string }; thinkingLevel?: ThinkingLevel } | null
>;
/** GET /api/models response. */
export interface ModelCatalogResponse {
models: ModelCatalogEntry[];
defaults: CatalogDefaults;
/** The conversation agents' resolved default — NULL when the
* conversation chain is empty (no factory floor; the turn gate
* refuses with 503 models_not_configured). */
defaults: CatalogDefaults | null;
/** The six background roles' resolved pins (same resolution the agent
* factories use — src/server/stores.ts effectiveSettings). */
* factories use — src/server/stores.ts effectiveSettings); NULL per
* role = chain empty. */
rolePins: RolePins;
/** The three slot tiers' resolved pins (model slots wave, 2026-09-20):
* same resolution chain — role override → slot → factory pin — reported
* at tier granularity; effectiveSettings().models verbatim. */
/** The three slot tiers' resolved pins: role override → slot at tier
* granularity; effectiveSettings().models verbatim, NULL per unset
* tier. */
slots: SlotPins;
/** Providers with a PRESENT key, and which env var gates each — the
* model slots wave's federation surface (2026-09-20): the picker lists
src/core/settings-resolution.ts
+82 -88
View File
@@ -5,17 +5,17 @@
* delta-walking of their own.
*
* Resolution rule (owner decision): explicit value in the record wins;
* absent key resolves to the compiled factory default. Model-role defaults
* come from config.ts's MODELS, threaded in as a parameter — this module
* must stay browser-safe/pure, so it imports config TYPES only, never the
* runtime module (config.ts pulls node:path). The composition root
* (src/server/stores.ts) owns the glue: effectiveSettings() reads the
* record through the store and passes MODELS here.
* absent key resolves to the compiled factory default — EXCEPT MODELS
* (BYOK all the way down, owner 2026-09-22): no factory pins exist, so a
* model chain that ends empty resolves to null/undefined and whatever
* needs it refuses honestly (503 models_not_configured). This module
* stays browser-safe/pure, so it imports config TYPES only, never the
* runtime module (config.ts pulls node:path).
*
* Deliberately NOT resolved here (see settings.ts module doc):
* - per-chat overrides: the chat record's ChatSettings applies via
* piEngineFor(role, chatRecord.settings) exactly as before — prober and
* teacher stay chat-controlled, never settings-record-controlled;
* - per-chat overrides: the chat record's ChatSettings applies via the
* conversation chain in chat-handler (chat.model → strong slot) — prober
* and teacher stay chat-controlled, never settings-record-controlled;
* - tool toggles: passed through as the delta map; consumers look up by
* tool id with absent → enabled (the honoring lives in the agent band);
* - prompt switches: passed through; effective composition lives in
@@ -42,27 +42,24 @@ import {
type UserSettings,
} from "./settings.ts";
/** The factory layer for model roles — config.ts's MODELS shape (types only
* here; the runtime object is threaded in by the caller). Model slots
* (2026-09-20): the factory map spans ALL nine roles (the strong slot's
* factory fallback is the prober pin), while per-role OVERRIDES stay
* background-only (the conversation pair is chat-settings-controlled). */
export type ModelRoleSpecs = Record<ModelRole, { provider: string; model: string; effort: ThinkingLevel }>;
/** A fully-resolved model slot: the tier's concrete model (slot over factory
* rep pin) — strong's factory rep is the prober pin, cheap's the researcher
* pin, vision's the visualMaker pin (owner-settled tier mapping). */
/** A configured model slot: the tier's concrete model, with its optional
* thinking level. UNSET level is legal — the engine build resolves it
* from config.ts's DEFAULT_THINKING_ORDER (first entry the model
* supports; owner, 2026-09-22). A slot is CONFIGURED iff its model is
* present; a level-only slot is inert (kept on disk, resolved as null). */
export interface EffectiveSlot {
provider: string;
id: string;
thinkingLevel: ThinkingLevel;
thinkingLevel?: ThinkingLevel;
}
/** A fully-resolved model pin for one background role. */
/** A fully-resolved model pin for one role — same shape/level semantics
* as EffectiveSlot; undefined means the role's whole chain came up
* empty and whatever needs it refuses honestly. */
export interface EffectiveModelSpec {
provider: string;
id: string;
thinkingLevel: ThinkingLevel;
thinkingLevel?: ThinkingLevel;
}
/** The concrete appearance view — same shape as the factory's. */
@@ -75,7 +72,7 @@ export interface EffectiveSettings {
* ({} = no default = BYOK "no model" first boot). */
chat: ChatSettings;
agent: {
models: Record<BackgroundModelRole, EffectiveModelSpec>;
models: Record<BackgroundModelRole, EffectiveModelSpec | undefined>;
/** Delta map as stored — consumers resolve absent key → enabled. */
tools: Record<string, boolean>;
/** Delta map as stored — absent key → override off. */
@@ -85,84 +82,81 @@ export interface EffectiveSettings {
* lives in server/search.ts, where keys are visible. */
search: { engine?: SearchEngineId };
};
/** The resolved strong/cheap/vision slots (model slots wave,
* 2026-09-20): each tier's concrete model (slot over factory rep pin)
* — the per-role chain (override → slot → pin) is resolved into
* agent.models; these are the tier-level views the picker captions. */
models: Record<ModelSlotName, EffectiveSlot>;
/** The resolved strong/cheap/vision slots: each tier's concrete model or
* NULL when the tier holds no model (no factory floor — BYOK all the
* way down, owner 2026-09-22). These are the tier-level views the
* picker captions and the per-need refusal gates read. */
models: Record<ModelSlotName, EffectiveSlot | null>;
appearance: EffectiveAppearance;
graph: Required<GraphSettings>;
}
/** Full-chain resolution for ANY role (model slots wave, 2026-09-20):
* per-role override (background roles only) → slot → factory pin. The
* conversation pair (prober/teacher) resolves the same chain MINUS the
* role override (their override is the chat's own settings — the
* chat-handler overlays it itself). This is the engine wiring's one call. */
export function resolveRoleSpec(
role: ModelRole,
record: UserSettings | null,
factory: ModelRoleSpecs,
): EffectiveModelSpec {
const bg = BACKGROUND_MODEL_ROLES as readonly string[];
const override = bg.includes(role) ? record?.agent?.models?.[role as BackgroundModelRole] : undefined;
return resolveModelSpec(role as BackgroundModelRole, override, factory, record?.models);
}
/** The factory representative pin for a slot tier — the pin the slot
* falls back to when unset (owner-settled mapping: strong's rep is the
* prober pin — prober and teacher share the strong tier — cheap's the
* researcher pin, vision's the visualMaker pin). */
export function factorySlotRep(slot: ModelSlotName, factory: ModelRoleSpecs): EffectiveSlot {
const rep =
slot === "strong" ? factory.prober : slot === "cheap" ? factory.researcher : factory.visualMaker;
// The factory contract spans all nine roles (stores.ts MODEL_PINS) —
// the rep lookup cannot miss; the cast is for the checker only.
const r = rep as { provider: string; model: string; effort: ThinkingLevel };
return { provider: r.provider, id: r.model, thinkingLevel: r.effort };
}
/** Resolve one slot: the slot override per field, the factory rep pin per
* absent field (model slots wave, 2026-09-20). */
export function resolveSlot(
slot: ModelSlotName,
override: ModelSlot | undefined,
factory: ModelRoleSpecs,
): EffectiveSlot {
const rep = factorySlotRep(slot, factory);
/** The conversation pair's chain (BYOK, owner 2026-09-22): the chat's own
* model override over the STRONG slot, each field independent (a bare
* chat thinking level rides the strong slot's model). null = no model
* anywhere — the turn gate refuses with the 503 models_not_configured
* envelope. */
export function resolveConversationSpec(
chat: ChatSettings | undefined,
strong: EffectiveSlot | null,
): EffectiveModelSpec | null {
const model = chat?.model ?? (strong ? { provider: strong.provider, id: strong.id } : undefined);
if (!model) return null;
return {
provider: override?.model?.provider ?? rep.provider,
id: override?.model?.id ?? rep.id,
thinkingLevel: override?.thinkingLevel ?? rep.thinkingLevel,
provider: model.provider,
id: model.id,
thinkingLevel: chat?.thinkingLevel ?? strong?.thinkingLevel,
};
}
/** Resolve one background role's spec — the model slots chain (owner
* decision 2026-09-20): role override → SLOT → factory pin, each field
* independent (a role can borrow only the slot's model, or only its
* thinking budget). An absent provider coordinate anywhere resolves to
* the factory pin's provider (the BYOK default, openrouter today) —
* legacy ids without a provider ride the factory provider. */
/** Full-chain resolution for ANY role (BYOK, owner 2026-09-22):
* per-role override (background roles only) → SLOT → NOTHING. The
* conversation pair (prober/teacher) resolves the same chain MINUS the
* role override (their override is the chat's own settings — the
* chat-handler overlays it itself). undefined = the chain came up
* empty; the per-need refusal gates fire on exactly that. This is the
* engine wiring's one call. */
export function resolveRoleSpec(role: ModelRole, record: UserSettings | null): EffectiveModelSpec | undefined {
const bg = BACKGROUND_MODEL_ROLES as readonly string[];
const override = bg.includes(role) ? record?.agent?.models?.[role as BackgroundModelRole] : undefined;
return resolveModelSpec(role, override, record?.models);
}
/** Resolve one slot to its configured view, or null when the tier holds
* no model (a thinkingLevel-only slot is inert — the level has nothing
* to ride without the factory floor). */
export function resolveSlot(slot: ModelSlot | undefined): EffectiveSlot | null {
if (!slot?.model) return null;
return { provider: slot.model.provider, id: slot.model.id, thinkingLevel: slot.thinkingLevel };
}
/** Resolve one role's spec — the BYOK chain (owner 2026-09-22): role
* override → SLOT → NOTHING, each field independent (an override may
* borrow only the slot's model, or only its thinking budget — a level
* with no model anywhere stays unset and the engine build resolves it
* from DEFAULT_THINKING_ORDER). undefined = no model resolved; the
* per-need refusal gates fire on exactly that. */
export function resolveModelSpec(
role: BackgroundModelRole,
role: ModelRole,
override: RoleModelOverride | undefined,
factory: ModelRoleSpecs,
slots?: Partial<Record<ModelSlotName, ModelSlot>>,
): EffectiveModelSpec {
// The factory contract spans all nine roles; background roles always hit.
const spec = factory[role] as { provider: string; model: string; effort: ThinkingLevel };
): EffectiveModelSpec | undefined {
const slotName = ROLE_SLOT[role] as ModelSlotName | undefined;
const slot = slotName ? slots?.[slotName] : undefined;
const model = override?.model ?? slot?.model;
if (!model) return undefined;
return {
provider: override?.model?.provider ?? slot?.model?.provider ?? spec.provider,
id: override?.model?.id ?? slot?.model?.id ?? spec.model,
thinkingLevel: override?.thinkingLevel ?? slot?.thinkingLevel ?? spec.effort,
provider: model.provider,
id: model.id,
thinkingLevel: override?.thinkingLevel ?? slot?.thinkingLevel,
};
}
/** Resolve the whole record. `record` may be null (first boot) — every
* field then lands on its factory default. */
export function resolveSettings(record: UserSettings | null, factory: ModelRoleSpecs): EffectiveSettings {
* factory-defaulted field lands on its default; MODEL fields resolve to
* null/undefined when unset (no factory pins exist — BYOK all the way
* down, owner 2026-09-22). */
export function resolveSettings(record: UserSettings | null): EffectiveSettings {
const agent = record?.agent ?? {};
const modelsSection = record?.models ?? {};
const appearance = record?.appearance ?? {};
@@ -170,9 +164,9 @@ export function resolveSettings(record: UserSettings | null, factory: ModelRoleS
const graph = record?.graph ?? {};
const factoryAppearance = FACTORY_SETTINGS.appearance;
const factoryMd = factoryAppearance.mdRendering;
const models = {} as Record<BackgroundModelRole, EffectiveModelSpec>;
const models = {} as Record<BackgroundModelRole, EffectiveModelSpec | undefined>;
for (const role of BACKGROUND_MODEL_ROLES) {
models[role] = resolveModelSpec(role, agent.models?.[role], factory, modelsSection);
models[role] = resolveModelSpec(role, agent.models?.[role], modelsSection);
}
return {
general: {
@@ -186,9 +180,9 @@ export function resolveSettings(record: UserSettings | null, factory: ModelRoleS
search: agent.search ?? {},
},
models: {
strong: resolveSlot("strong", modelsSection.strong, factory),
cheap: resolveSlot("cheap", modelsSection.cheap, factory),
vision: resolveSlot("vision", modelsSection.vision, factory),
strong: resolveSlot(modelsSection.strong),
cheap: resolveSlot(modelsSection.cheap),
vision: resolveSlot(modelsSection.vision),
},
appearance: {
mode: appearance.mode ?? factoryAppearance.mode,
src/core/settings.ts
+51 -14
View File
@@ -5,8 +5,10 @@
*
* DELTA SEMANTICS (owner decision, 2026-09-19): the record stores EXPLICIT
* user choices only — ABSENT keys resolve to the factory defaults compiled
* into this module (FACTORY_SETTINGS) and config.ts's MODELS (the compiled
* factory layer for model roles). A software update that ADDS a knob (new
* into this module (FACTORY_SETTINGS). MODELS are the exception (owner,
* 2026-09-22 — BYOK all the way down): no factory pins exist, so an absent
* model key resolves to EMPTY and the per-need gates refuse honestly. A
* software update that ADDS a knob (new
* tool, new role, new appearance field) therefore needs no migration: old
* records simply lack the new keys. `version` is a PRIVATE schema counter
* for BREAKING shape changes only (field renamed / meaning changed) —
@@ -26,6 +28,7 @@
*/
import type { ThinkingLevel } from "@earendil-works/pi-agent-core";
import type { ModelRole } from "../../config.ts";
import type { ChatSettings } from "./contracts/chat.ts";
import type { SearchEngineId } from "./search.ts";
@@ -34,9 +37,8 @@ import type { SearchEngineId } from "./search.ts";
export const CURRENT_SETTINGS_VERSION = 1;
/** The background-agent roles whose models the Agent settings expose.
* Derived from config.ts's MODELS minus the conversation pair — prober and
* teacher stay chat-settings-controlled (one controller, the model picker).
* Keep in sync with config.ts's MODELS (tests pin the set). */
* The conversation pair (prober/teacher) stays out — chat-controlled via
* the strong slot (one controller, the model picker). Tests pin the set. */
export const BACKGROUND_MODEL_ROLES = [
"researcher",
"demoBuilder",
@@ -67,9 +69,44 @@ export interface RoleModelOverride {
export const MODEL_SLOTS = ["strong", "cheap", "vision"] as const;
export type ModelSlotName = (typeof MODEL_SLOTS)[number];
/** One slot's override — same per-field independence as a role override:
* absent model keeps the tier's current model, absent thinkingLevel keeps
* its thinking budget. */
/** The machine-readable NOT-CONFIGURED refusal envelope (owner, 2026-09-22):
* every per-need gate — the turn gate in chat-handler, a job spawn, a
* notes import — refuses with this exact shape. HTTP callers get 503 +
* this JSON; in-band consumers (the job record's error field) carry the
* human `error` line. `missing` names the slot TIER(S) the refused
* action needs and found unset. */
export const MODELS_NOT_CONFIGURED_CODE = "models_not_configured" as const;
export interface ModelsNotConfigured {
/** Human-readable — safe to render as-is. */
error: string;
/** The machine discriminator the client switches on. */
code: typeof MODELS_NOT_CONFIGURED_CODE;
/** Which slot tiers the refused action needs ("strong" | "cheap" | "vision"). */
missing: ModelSlotName[];
}
export function modelsNotConfigured(missing: ModelSlotName[]): ModelsNotConfigured {
return {
error: "No model configured — set up your models first",
code: MODELS_NOT_CONFIGURED_CODE,
missing,
};
}
/** The typed throw for in-process seams (extraction pipeline): callers
* catch it to skip honestly (trigger) or surface the 503 envelope
* (routes) instead of logging a failure. */
export class ModelsNotConfiguredError extends Error {
readonly envelope: ModelsNotConfigured;
constructor(missing: ModelSlotName[]) {
super(modelsNotConfigured(missing).error);
this.name = "ModelsNotConfiguredError";
this.envelope = modelsNotConfigured(missing);
}
}
/** One slot's stored state. BYOK (owner, 2026-09-22): the WRITE side
* requires model (a level alone can never run — no factory floor); the
* sanitizer keeps whatever is on disk, but RESOLUTION treats a slot
* without a model as unset (inert level, no phantom default). */
export interface ModelSlot {
/** The model swap: both coordinates or neither. */
model?: { provider: string; id: string };
@@ -99,7 +136,7 @@ export function slotsConfigured(models: ModelsSettings | undefined): boolean {
/** Which slot a role draws from (model slots wave, 2026-09-20). Includes the
* conversation pair and pdfReader (not BackgroundModelRoles — the slot
* tier spans every role; only background roles surface per-role overrides). */
export const ROLE_SLOT: Readonly<Record<string, ModelSlotName>> = {
export const ROLE_SLOT: Readonly<Record<ModelRole, ModelSlotName>> = {
prober: "strong",
teacher: "strong",
researcher: "cheap",
@@ -201,11 +238,11 @@ export interface UserSettings {
[key: string]: unknown;
}
/** Factory defaults for everything core can pin (owner decisions 2026-09-19).
* Model-role defaults live in config.ts's MODELS (the compiled factory
* layer); the chat section has NO factory model (BYOK first boot doctrine
* — "no model" until the user picks one); tools default on; prompts
* default off — hence their absence here. */
/** Factory defaults for everything core can pin (owner decisions
* 2026-09-19). NO factory models exist anywhere (owner, 2026-09-22 — BYOK
* all the way down): the chat section and every slot stay empty until the
* user picks, and the per-need gates refuse honestly; tools default on;
* prompts default off — hence their absence here. */
export interface FactorySettings {
version: number;
general: Required<GeneralSettings>;