amosblomqvist
d047e7b2dc
Every notes chunk and chat coverage batch is its own job — registry row,
usage record, failure domain. Extracts fan out process-wide behind a
3-permit extract semaphore on globalThis (new 'queued' phase); audits and
merges stay serialized behind the existing FIFO gate ('waiting'), so each
auditor explores the graph after every earlier job's merge.
- ExtractionPhase gains 'queued'; RunningExtraction gains part {i,n};
the feed shows one row per job with a '· i/N' suffix (status route)
- abortExtraction is source-wide: every job of a fan-out dies together;
merged jobs keep their coverage checkpoints
- failure isolation at the fan-out (Promise.allSettled + aggregation):
a failed chunk no longer kills its siblings — the run returns what
actually merged; job-level aborts log, pre-fan-out errors still throw
- usage ledgers: one record per job (source-keyed), failed jobs bill
their partial spend
- runJob/aggregateJobs helpers; setPhase/recordStep/extract/audit/
notesExtract key the registry by jobKey (src#i) — notesExtract's
signal lookup was the one real bug the new tests caught (source-id
lookup meant fan-out aborts never reached the model call)
- semaphore is handoff-correct (fresh takes increment; handed-off
permits don't double-count — an inflated counter deadlocks at 3)
- tests: isolation pins rewritten (stage failure now resolves zeros),
single-flight/registry tails updated, per-job usage counts, and a new
describe pinning the fan-out: 3 extracting + 5 queued, parts 1..8,
source-wide abort, per-job usage on failure, extract peak == 3,
audit peak == 1
- docs/architecture.md: serialized-audit paragraph rewritten for
per-chunk jobs; two-front-doors wording aligned
32 lines
1.3 KiB
TypeScript
32 lines
1.3 KiB
TypeScript
/**
|
|
* GET /api/graph/status — the in-flight background extraction runs
|
|
* (vision §1: the pipeline status must be visible on the home graph).
|
|
* Thin transport over the pipeline's module-level registry: phase
|
|
* (queued / extracting / waiting / auditing / merging) per job, oldest
|
|
* first — a multi-chunk source shows one row PER CHUNK ("· i/N" suffix,
|
|
* per-chunk jobs, owner 2026-09-28). Titles joined from the chat store;
|
|
* a missing chat record still reports the run.
|
|
*/
|
|
import { chatStore, extractionStatuses } from "@/src/server";
|
|
|
|
export const dynamic = "force-dynamic";
|
|
|
|
export async function GET() {
|
|
const runs = extractionStatuses().map((run) => ({
|
|
chatId: run.chatId,
|
|
// The run's own label (the file path for notes imports) wins; chat
|
|
// runs fall back to the chat store's title. Chunk jobs append their
|
|
// part ("· 2/5") — one row per chunk of the fan-out.
|
|
title:
|
|
(run.label ?? chatStore.getChat(run.chatId)?.title ?? run.chatId) +
|
|
(run.part ? ` · ${run.part.i}/${run.part.n}` : ""),
|
|
phase: run.phase,
|
|
startedAt: run.startedAt,
|
|
// Live per-stage usage (the money channel): model + per-step CallUsage,
|
|
// accumulated as steps bill. Cost and context-window folds are the
|
|
// consumer's (core/prices.ts helpers — nothing stores dollars).
|
|
usage: run.usage,
|
|
}));
|
|
return Response.json({ runs });
|
|
}
|