amosblomqvist
f0267a5fd9
Core-survival labelings (vision section 1, DECIDED 2026-09-13): ClusterLabeling records in a new append-only ledger (data/graph/cluster-labels.jsonl), immutable, id = clu_ + sha256 of the sorted core so re-naming races and re-runs dedupe at the store seam. matchClusterLabelings (core/clusters.ts, pure): a community keeps a record's label while at least half its core survives in it; both sides single-successor (a split keeps one name; greedy by overlap, ties to the newer record). The namer (src/agent/subagents/cluster-namer.ts): ONE singular cheap LLM call (new 'labeler' role, glm-flash/low) names every new/changed community at once, given member names + statements (capped at 12 members per cluster). refreshClusterLabels runs POST-EXTRACTION (user-decided) from both front doors, fire-and-forget — naming failure never fails the run; refreshes serialize behind a chain so parallel imports never mint duplicate labels. import-notes.ts awaits clusterNamingSettled before exit. Payload: GraphSnapshot.clusters (>= MIN_CLUSTER_SIZE 3) joins detection with labelings at read time — claimed clusters carry the record's label and id; unlabeled ones carry a member-hash id and null label. Detection is deterministic so no memo in v1 (ms at current scale; revisit at vault scale). Tests (372/372): matcher half-rule/split/merge/greedy/tie cases; naming end-to-end through importNotes (one call, one record, claim on re-refresh), naming failure non-fatal; snapshot derivation with labeling join; 11 test pipelines now pin an explicit labeler seam.
69 lines
2.8 KiB
TypeScript
69 lines
2.8 KiB
TypeScript
/**
|
|
* Import raw notes files through the format-agnostic front door (vision §1,
|
|
* 2026-09-13) — every *.md / *.txt file in a directory becomes one
|
|
* extraction run (extractor → auditor → merger) over its raw text. Run
|
|
* from this worktree only — data never crosses trees:
|
|
*
|
|
* npx tsx --env-file=.env.local scripts/import-notes.ts <dir> [concurrency]
|
|
*
|
|
* Default dir: real-data-extraction/learn-sessions (the prototype
|
|
* learning-session exports — deliberately NOT imported as chats: they run
|
|
* through the notes path, raw). Runs in this process: the dev server's
|
|
* status chips do NOT see them (registry is per-process) — the dev server
|
|
* sees runs triggered through /api/graph/import.
|
|
*/
|
|
import * as fs from "node:fs";
|
|
import * as path from "node:path";
|
|
import { createExtractionPipeline } from "../src/agent/subagents/extraction.ts";
|
|
import { clusterNamingSettled } from "../src/agent/subagents/cluster-namer.ts";
|
|
import { getGraph } from "../src/core/graph.ts";
|
|
import { stores } from "../src/server/stores.ts";
|
|
|
|
async function main() {
|
|
const dir = process.argv[2] ?? "real-data-extraction/learn-sessions";
|
|
const limit = Number(process.argv[3] ?? 4);
|
|
const files = fs
|
|
.readdirSync(dir)
|
|
.filter((f) => f.endsWith(".md") || f.endsWith(".markdown") || f.endsWith(".txt"))
|
|
.sort();
|
|
console.log(`[import-notes] ${files.length} note files from ${dir} (concurrency ${limit})`);
|
|
|
|
const pipeline = createExtractionPipeline(stores);
|
|
const queue = [...files];
|
|
const totals = { nodes: 0, edges: 0, events: 0, failed: 0 };
|
|
const workers = Array.from({ length: Math.min(limit, files.length) }, async () => {
|
|
for (;;) {
|
|
const file = queue.shift();
|
|
if (!file) return;
|
|
const filePath = path.join(dir, file);
|
|
const t0 = Date.now();
|
|
try {
|
|
const stats = await pipeline.importNotes({ filePath, text: fs.readFileSync(filePath, "utf-8") });
|
|
totals.nodes += stats.nodes;
|
|
totals.edges += stats.edges;
|
|
totals.events += stats.events;
|
|
console.log(
|
|
`[import-notes] ${file}: ${stats.nodes} nodes, ${stats.edges} edges, ${stats.events} events in ${((Date.now() - t0) / 1000).toFixed(1)}s`,
|
|
);
|
|
} catch (err) {
|
|
totals.failed++;
|
|
console.error(`[import-notes] ${file} FAILED:`, err);
|
|
}
|
|
}
|
|
});
|
|
await Promise.all(workers);
|
|
|
|
// The post-merge cluster naming is fire-and-forget inside the pipeline;
|
|
// settle it before the process exits so labels actually land.
|
|
await clusterNamingSettled();
|
|
|
|
const graph = getGraph(stores.graphStore);
|
|
const labels = stores.graphStore.readClusterLabels().length;
|
|
console.log(
|
|
`[import-notes] done: +${totals.nodes} nodes, +${totals.edges} edges, +${totals.events} events, ${totals.failed} failed — ` +
|
|
`graph now ${graph.nodes.length} nodes, ${graph.edges.length} edges, ${graph.events.length} events, ` +
|
|
`${labels} named clusters`,
|
|
);
|
|
}
|
|
|
|
main(); |