Semantic topic clusters: local embeddings replace Louvain

Louvain only ever found the reasoning chains (the graph is disconnected
by nature — every connected component became its own cluster) and
edge-orphans could never join anything. Clusters are now RELATED TOPICS:
propositions grouped by meaning.

- Embedder (src/server/embeddings.ts): in-process transformers.js v4
  running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download),
  MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs,
  L2-normalized so cosine = dot. Model/dim mismatch voids the ledger
  (full re-embed). Injected at the composition root — the agent layer
  never imports the server layer (architecture law).
- Ledger (data/graph/embeddings.json): model + dim + vectors + the
  persisted cluster partition. Reclustering runs at the END of every
  embedding batch (post-merge hook, both front doors) — never per read
  (the agglomerative pass is ~1.5s at 338 nodes).
- Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage
  agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers
  stay singletons by default; attachStragglers is the all-in toggle.
  Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans
  land in real topics, 10 stragglers.
- Consumers (snapshot, settle, namer, replay) read the persisted
  partition — one cluster concept everywhere; core-survival label
  matching unchanged (20/26 topics kept labels).
- Louvain retired: detectCommunities + graphology* deps deleted
  (graphology-louvain had leaked into the client bundle since the
  clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed.
- scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s
  including the one-time model download).
This commit is contained in:
amosblomqvist
2026-09-13 14:29:16 +03:00
parent 3a1c18b9de
commit f5edc7f0a5
28 changed files with 1443 additions and 431 deletions
+8 -7
View File
@@ -1,5 +1,4 @@
import type { GraphEdge, GraphNode } from "./contracts.ts";
import { detectCommunities } from "./clusters.ts";
import type { EmbeddingsLedger, GraphEdge, GraphNode } from "./contracts.ts";
import { incrementalLayout } from "./layout.ts";
import type { Position } from "./contracts.ts";
@@ -13,7 +12,7 @@ import type { Position } from "./contracts.ts";
*
* SPLIT FROM layout.ts deliberately (2026-09-13): layout.ts is in the
* browser barrel, and the settle needs the Louvain partition for the
* cluster centroid pull — graphology-louvain must not ship in the client
* cluster centroid pull — the heavy deps must not ship in the client
* bundle. Server code imports this module DIRECTLY; it is never
* re-exported through src/core/index.ts.
*/
@@ -22,6 +21,7 @@ export function settleGraphLayout(graphStore: {
readEdges(): GraphEdge[];
readPositions(): Record<string, Position>;
writePositions(positions: Record<string, Position>): void;
readEmbeddings(): EmbeddingsLedger;
}): void {
const nodes = graphStore.readNodes();
if (nodes.length === 0) return;
@@ -34,10 +34,11 @@ export function settleGraphLayout(graphStore: {
});
const edges = graphStore.readEdges();
const previous = new Map(Object.entries(graphStore.readPositions()));
// The cluster partition for the centroid pull: same Louvain the
// snapshot uses (same nodes/edges = same communities). 338 nodes:
// ~1-2ms, noise next to the settle itself.
const { communities } = detectCommunities(unique, edges);
// The cluster partition for the centroid pull: the SAME persisted
// semantic topics the snapshot renders (computed by the embedder
// once per batch, stored in the embedding ledger) — one cluster
// concept everywhere.
const communities = graphStore.readEmbeddings().clusters;
const next = incrementalLayout(unique, edges, previous, { clusters: communities });
graphStore.writePositions(Object.fromEntries(next));
}