sorokin.oleg.edu/learnharness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Wave 7 chunk B: remove the embeddings machinery — the ladder collapses

Browse Source 
The owner's wave-7 ruling: embeddings die entirely (git history is the
stash); clustering is birth-session derived (chunk A, 950ddfd), labels
are session names, and the prober's graph tools replace injection.

- settings: GraphSettings collapses to { enabled } — embedding,
  embeddingModel, clustering, labeling all die. Migration story: stale
  record keys are UNREAD and dropped by the sanitizer's reconstruction;
  the next graph-section PUT (replace-whole) persists the collapsed
  shape. The settings-panel GraphSection renders one master row.
- deleted: src/server/embeddings.ts (adapters, resolveEmbeddingPortFor,
  embedNewNodes, embeddingPort, embeddingModelCatalog), the
  GET /api/models/embeddings route, scripts/embed-graph.ts, tests for
  all three, and the /api/models/embeddings picker surfaces in the UI.
- prices: EMBEDDING_PRICES overlay + callCost fallback die. cost-summary:
  the embedding stage fold dies — back to three stages (extraction/
  audit/naming) per the owner ruling; historical stages.embedding spend
  in old run records is simply not displayed.
- ports: EmbeddingPort/EmbeddingResult/EmbeddingTask die. The embeddings
  LEDGER surface (readEmbeddings/writeEmbeddings + EmbeddingsLedger +
  EMBEDDINGS_FILE + graph-deletion's vector pruning) STAYS — inert data,
  kept only so graph-context.ts compiles until Teacher's pass removes
  it; the seam dies in the follow-up interlude.
- chat-handler (approved cross-lane unwiring, wave-6 precedent): the
  extraction pipeline's embedder becomes a documented no-op (nothing
  embedded, nothing spent, ledger untouched) until Teacher's pass
  removes the phase; the graph-context injection dispatch gate, the
  graphContext deps wiring, the setContextRespawn seam, and
  turn-service's triggerGraphContext are gone — injection is dead
  code pending Teacher's file deletions.
- scripts: backfill-graph/import-notes pass the same no-op embedder;
  replay-layout derives clusters from birthSessionPartition (the same
  partition the real settle uses) instead of the embeddings ledger.
- settings-body: parseGraphSettingsBody reads the master switch only —
  old-shape keys in a body are ignored, not rejected; bad shapes 400.
- next.config: the @huggingface tracing-exclusion guards die with the
  last reference.

Cross-lane flags: extraction.ts/graph-context.ts keep stale embed
comments + the embed phase until Teacher's full pass (their parallel
interlude owns those files — zero overlap kept by design);
semantic-clusters.ts + its test stay one more round (extraction.test.ts,
Teacher's file, still imports it — they delete it with the namer cases);
ChatSettings.graphContextTokens + the graph-context-control UI stay
until Teacher+UI.

Gates: tsc clean, eslint clean, 885/885 tests, build compiles.
Live-verified on the port-3004 dev tree: PUT /api/settings/graph with
dead keys collapses on write, bad shapes 400, GET reads {enabled: true},
/api/models/embeddings 404s.
This commit is contained in:
amosblomqvist
2026-09-23 15:02:55 +03:00
parent 0fcffdffaf
commit c86430e8bf
26 changed files with 120 additions and 1823 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/backfill-graph.ts
+1 -2
View File
@@ -8,14 +8,13 @@
import { chatStore } from "../src/store/chats.ts";
import { graphStore } from "../src/store/graph.ts";
import { createExtractionPipeline } from "../src/agent/subagents/extraction.ts";
import { embedNewNodes } from "../src/server/embeddings.ts";
import { getGraph } from "../src/core/graph.ts";
async function main() {
const only = process.argv[2]; // optional: extract just one chat id
const chats = only ? chatStore.listChats().filter((c) => c.id === only) : chatStore.listChats();
console.log(`[backfill] ${chats.length} chats to extract`);
const pipeline = createExtractionPipeline({ chatStore, graphStore, embedder: embedNewNodes });
const pipeline = createExtractionPipeline({ chatStore, graphStore, embedder: async () => ({ embedded: 0, total: 0, skipped: "embeddings removed (wave 7)" }) });
for (const chat of chats) {
const t0 = Date.now();
try {
scripts/embed-graph.ts
-47
View File
@@ -1,47 +0,0 @@
/**
* Embed the whole graph (backfill / cold start / model change): one-time
* pass that fills the embedding ledger for every node — the same
* embedNewNodes the post-merge hook runs, just with an empty (or
* stale-model) ledger so every node is "new".
*
* BYOK (wave 6, 2026-09-23): vectors come from the configured provider
* (graph.embeddingModel, keys from the auth store) — this script resolves
* the live state up front so an unconfigured machine says so before
* anything bills. A model CHANGE re-embeds everything (ledger staleness).
*
* npx tsx scripts/embed-graph.ts
*/
import { embedNewNodes, embeddingPort } from "../src/server/embeddings.ts";
import { graphStore } from "../src/store/graph.ts";
async function main() {
const resolved = embeddingPort();
if (!resolved.ok) {
console.error(`[embed] cannot run — ${resolved.reason}`);
process.exit(1);
}
const t0 = Date.now();
console.log(`[embed] model: ${resolved.port.model} (provider embeddings, batch 32)`);
const { embedded, total, usage, failed } = await embedNewNodes(graphStore);
const secs = ((Date.now() - t0) / 1000).toFixed(1);
if (usage) {
const tokens = usage.steps.reduce((sum, s) => sum + (s.inputTokens ?? 0), 0);
console.log(`[embed] embedding spend: ${tokens} prompt tokens over ${usage.steps.length} call(s)`);
}
// Usage-on-error + pay-once (owner ruling 2026-09-23): a mid-run provider
// failure still reports the landed batches' spend above; the landed
// vectors are persisted, so the honest count is what LANDED — the
// in-memory `total` would read as a ledger claim it can't make.
if (failed) {
console.error(
`[embed] failed after ${embedded} node(s) landed (${secs}s) — the ledger keeps exactly those (a retry embeds only the rest): ${failed}`,
);
process.exit(1);
}
console.log(`[embed] embedded ${embedded} nodes → ledger holds ${total} vectors (${secs}s)`);
}
main().catch((err) => {
console.error("[embed] failed:", err);
process.exit(1);
});
scripts/import-notes.ts
+1 -2
View File
@@ -15,7 +15,6 @@
import * as fs from "node:fs";
import * as path from "node:path";
import { createExtractionPipeline } from "../src/agent/subagents/extraction.ts";
import { embedNewNodes } from "../src/server/embeddings.ts";
import { clusterNamingSettled } from "../src/agent/subagents/cluster-namer.ts";
import { getGraph } from "../src/core/graph.ts";
import { stores } from "../src/server/stores.ts";
@@ -29,7 +28,7 @@ async function main() {
.sort();
console.log(`[import-notes] ${files.length} note files from ${dir} (concurrency ${limit})`);
const pipeline = createExtractionPipeline({ ...stores, embedder: embedNewNodes });
const pipeline = createExtractionPipeline({ ...stores, embedder: async () => ({ embedded: 0, total: 0, skipped: "embeddings removed (wave 7)" }) });
const queue = [...files];
const totals = { nodes: 0, edges: 0, events: 0, failed: 0 };
const workers = Array.from({ length: Math.min(limit, files.length) }, async () => {
scripts/replay-layout.ts
+6 -9
View File
@@ -18,6 +18,7 @@ const depthArg = process.argv.find((a) => a.startsWith("--depth="));
const depthMode: "off" | "vertical" | "radial" =
depthArg === "--depth=vertical" ? "vertical" : depthArg === "--depth=radial" ? "radial" : "off";
import { incrementalLayout } from "../src/core/layout.ts";
import { birthSessionPartition } from "../src/core/session-clusters.ts";
import type { GraphNode, NodeMapping, Position } from "../src/core/contracts.ts";
function main() {
@@ -32,12 +33,9 @@ function main() {
nodes.push(line);
}
const edges = graphStore.readEdges();
// The semantic-topic source: the embedding ledger (run
// scripts/embed-graph.ts first if missing/stale).
const embeddings = graphStore.readEmbeddings();
if (Object.keys(embeddings.vectors).length === 0) {
console.warn("[replay] embedding ledger is empty — clusters will be absent (run scripts/embed-graph.ts)");
}
// The cluster source (wave 7): the birth-session partition, derived on
// read — same as the real settle (core/layout-settle.ts).
const birthClusters = birthSessionPartition(nodes);
// Batches: each imported file, in real import order (sources.json
// importedAt). A node belongs to the file whose mapping first claims it
@@ -85,9 +83,8 @@ function main() {
const batchNodes = nodes.filter((n) => delivered.has(n.id));
positions = incrementalLayout(batchNodes, activeEdges, positions, {
// The same partition the real settle would see for this batch:
// the persisted semantic topics, filtered to delivered nodes
// (never recomputed — the agglomerative pass is ~1.5s at scale).
clusters: embeddings.clusters
// the birth-session clusters, filtered to delivered nodes.
clusters: birthClusters
.map((c) => ({ members: c.members.filter((id) => delivered.has(id)) }))
.filter((c) => c.members.length > 0),
...(depthMode === "off"