amosblomqvist
d1b46132b9
Part 1 — the ledger + remnants interlude (master-dispatched, owner ruling 2026-09-23: nothing configures the graph beyond chat models): - The embedding ledger surface is gone: readEmbeddings/writeEmbeddings ports, the EmbeddingsLedger contract, EMBEDDINGS_FILE (config, store impls, graphRevision source list), and graph-deletion's vector-pruning step. On-disk data files stay untouched (inert history). - The extraction embedder seam dies wholesale: ExtractionPipelineDeps drops the embedder dep (chat-handler + backfill/import-notes stubs go with it), the 'embedding' phase leaves ExtractionPhase, the usage capture/persist helpers die, and stages.embedding leaves ExtractionRunUsage (historical records keep their shape — reads are lenient). Extraction is two stages again, like the local-embedder era. - Consumer-less remnants: latestQuizOutcomes + QUIZ_EVENT_KINDS, clusterContentId, matchClusterLabelings + Community + ClusterLabeling, the readClusterLabels/appendClusterLabel port seams + store impls + LEDGER_FILES.clusterLabels. NamingRunUsage/readNamingRuns STAY (standing ruling — the cost pill renders historical naming spend); clusterHue + MIN_CLUSTER_SIZE stay (live). - The 'labeler' role dies: BACKGROUND_MODEL_ROLES, ROLE_SLOT, ModelRole, the catalog pin projections, and the UI roster strings. RolePins keys shrink with the union. The notes birth fallback (owner-approved 2026-09-23, surfaced by the interlude's live smoke: production notes imports create provenance-EMPTY nodes (importNotes passes messages: null — the file's anchor is the mapping line), so the provenance-only birth rule left every imported file unclustered and invisible to the prober's session list, against the locked file-birth ruling. A provenance-empty node now births under its FIRST mapping line's chatId (ledger order); only notes-born nodes ever hit the fallback (chat nodes always carry provenance; a re-encountered node's link-append re-opens the provenance rule). Deletion-safe: mapping lines of deleted chats are stripped alongside their provenance. Live-verified on port 3004: a fresh notes import clusters under its src_ id with the file path as the label, end to end with no embedder. 845/845 tests, typecheck/lint/build green. EOF )
37 lines
1.4 KiB
TypeScript
37 lines
1.4 KiB
TypeScript
/**
|
|
* One-time backfill: wipe the legacy ledgers and re-extract every chat with
|
|
* the redesigned pipeline (vision §1, commit b07c907). Run from the master
|
|
* tree only — data never crosses trees:
|
|
*
|
|
* npx tsx --env-file=.env.local scripts/backfill-graph.ts
|
|
*/
|
|
import { chatStore } from "../src/store/chats.ts";
|
|
import { graphStore } from "../src/store/graph.ts";
|
|
import { createExtractionPipeline } from "../src/agent/subagents/extraction.ts";
|
|
import { getGraph } from "../src/core/graph.ts";
|
|
|
|
async function main() {
|
|
const only = process.argv[2]; // optional: extract just one chat id
|
|
const chats = only ? chatStore.listChats().filter((c) => c.id === only) : chatStore.listChats();
|
|
console.log(`[backfill] ${chats.length} chats to extract`);
|
|
const pipeline = createExtractionPipeline({ chatStore, graphStore });
|
|
for (const chat of chats) {
|
|
const t0 = Date.now();
|
|
try {
|
|
const stats = await pipeline.runExtraction(chat.id);
|
|
console.log(
|
|
`[backfill] ${chat.id} (${chat.title.slice(0, 40)}): ` +
|
|
`${stats.nodes} nodes, ${stats.edges} edges, ${stats.events} events in ${((Date.now() - t0) / 1000).toFixed(1)}s`,
|
|
);
|
|
} catch (err) {
|
|
console.error(`[backfill] ${chat.id} FAILED:`, err);
|
|
}
|
|
}
|
|
const graph = getGraph(graphStore);
|
|
console.log(
|
|
`[backfill] done: ${graph.nodes.length} nodes, ${graph.edges.length} edges, ${graph.events.length} events`,
|
|
);
|
|
}
|
|
|
|
main();
|