amosblomqvist
2e9fd01c0c
Owner-locked extraction-UX redesign (dispatch 2026-09-24), part 1 (data
layer; the UI chunk follows).
THRESHOLD -> SETTING
- config.ts's EXTRACTION.chunkTokens dies; the default moves into
FACTORY_SETTINGS.graph.extractionChunkTokens (4096, the old constant's
value). GraphSettings gains the optional field, Required at resolution
via the standard ?? FACTORY pattern — pre-redesign records resolve the
default cleanly.
- FLOOR 500 (the implementer's call, documented on the field): every run
pays the graph-carrying prompt overhead out of the write channel's
money — a smaller budget fires penny-batches that cost more per
extracted token than the content is worth. The sanitizer CLAMPS up
(bd68486 clamp-never-reject discipline: fractions floor, junk drops);
the settings PUT REJECTS below-floor with the rationale in the message.
No ceiling: a large budget just delays extraction — the user's stated
cost/thoroughness tradeoff, not a misconfiguration to police.
- The trigger (turn-service.triggerExtraction) takes the budget as a
parameter — chat-handler passes effective.graph.extractionChunkTokens,
the fresh per-turn read, same as every other gate.
- The pipeline's BATCH side threads through the same number:
ExtractionPipelineDeps gains an OPTIONAL getChunkTokens getter with the
factory default (Teacher's 24 test fixtures stay untouched; the two
call-site lines inside extraction.ts are the only cross-band edits,
flagged). chat-handler + backfill/import-notes wire the live
effectiveSettings — trigger and batches share one budget, always.
PER-SESSION COVERAGE READ
- sessionCoverage(chatId) (server/chats.ts): the chat view's ring-meter
block — totalMessages/coveredCount/uncoveredCount/coveredTokens/
uncoveredTokens (the same coverageOf derivation /api/graph/coverage
serves, scoped to one chat), running (chat-kind extraction registered
for this id), and lastFoldedAt (the chat's newest coverage row's fold
— a cheap single pass; null before any fold). GET /api/chats/[id]
carries it as coverage: {...}; unknown chat -> null (the route 404s
first).
- /api/graph/coverage (the all-sessions list) compiles untouched — the
per-session read is a scoping of it, not a supersession; the UI's
sidebar/ring chunk decides which surface it consumes.
Tests: factory/sanitizer pins (clamp-up, fraction-floor, junk-drop),
resolution pins (absent -> 4096, stored 2000 wins), settings-body pins
(valid/absent/fraction/below-floor 400/junk 400), sessionCoverage pins
(derived split, newest-fold scoping, cross-session isolation, running via
the registry, null on unknown chat). 893/893.
66 lines
2.6 KiB
TypeScript
66 lines
2.6 KiB
TypeScript
/**
|
|
* Import raw notes files through the format-agnostic front door (vision §1,
|
|
* 2026-09-13) — every *.md / *.txt file in a directory becomes one
|
|
* extraction run (extractor → auditor → merger) over its raw text. Run
|
|
* from this worktree only — data never crosses trees:
|
|
*
|
|
* npx tsx --env-file=.env.local scripts/import-notes.ts <dir> [concurrency]
|
|
*
|
|
* Default dir: real-data-extraction/learn-sessions (the prototype
|
|
* learning-session exports — deliberately NOT imported as chats: they run
|
|
* through the notes path, raw). Runs in this process: the dev server's
|
|
* status chips do NOT see them (registry is per-process) — the dev server
|
|
* sees runs triggered through /api/graph/import.
|
|
*/
|
|
import * as fs from "node:fs";
|
|
import * as path from "node:path";
|
|
import { createExtractionPipeline } from "../src/agent/subagents/extraction.ts";
|
|
import { getGraph } from "../src/core/graph.ts";
|
|
import { effectiveSettings, stores } from "../src/server/stores.ts";
|
|
|
|
async function main() {
|
|
const dir = process.argv[2] ?? "real-data-extraction/learn-sessions";
|
|
const limit = Number(process.argv[3] ?? 4);
|
|
const files = fs
|
|
.readdirSync(dir)
|
|
.filter((f) => f.endsWith(".md") || f.endsWith(".markdown") || f.endsWith(".txt"))
|
|
.sort();
|
|
console.log(`[import-notes] ${files.length} note files from ${dir} (concurrency ${limit})`);
|
|
|
|
const pipeline = createExtractionPipeline({
|
|
...stores,
|
|
// Live budget: the graph section's setting, same as the app's trigger
|
|
getChunkTokens: () => effectiveSettings().graph.extractionChunkTokens,
|
|
});
|
|
const queue = [...files];
|
|
const totals = { nodes: 0, edges: 0, events: 0, failed: 0 };
|
|
const workers = Array.from({ length: Math.min(limit, files.length) }, async () => {
|
|
for (;;) {
|
|
const file = queue.shift();
|
|
if (!file) return;
|
|
const filePath = path.join(dir, file);
|
|
const t0 = Date.now();
|
|
try {
|
|
const stats = await pipeline.importNotes({ filePath, text: fs.readFileSync(filePath, "utf-8") });
|
|
totals.nodes += stats.nodes;
|
|
totals.edges += stats.edges;
|
|
totals.events += stats.events;
|
|
console.log(
|
|
`[import-notes] ${file}: ${stats.nodes} nodes, ${stats.edges} edges, ${stats.events} events in ${((Date.now() - t0) / 1000).toFixed(1)}s`,
|
|
);
|
|
} catch (err) {
|
|
totals.failed++;
|
|
console.error(`[import-notes] ${file} FAILED:`, err);
|
|
}
|
|
}
|
|
});
|
|
await Promise.all(workers);
|
|
|
|
const graph = getGraph(stores.graphStore);
|
|
console.log(
|
|
`[import-notes] done: +${totals.nodes} nodes, +${totals.edges} edges, +${totals.events} events, ${totals.failed} failed — ` +
|
|
`graph now ${graph.nodes.length} nodes, ${graph.edges.length} edges, ${graph.events.length} events`,
|
|
);
|
|
}
|
|
|
|
main(); |