Semantic topic clusters: local embeddings replace Louvain
Louvain only ever found the reasoning chains (the graph is disconnected by nature — every connected component became its own cluster) and edge-orphans could never join anything. Clusters are now RELATED TOPICS: propositions grouped by meaning. - Embedder (src/server/embeddings.ts): in-process transformers.js v4 running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download), MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs, L2-normalized so cosine = dot. Model/dim mismatch voids the ledger (full re-embed). Injected at the composition root — the agent layer never imports the server layer (architecture law). - Ledger (data/graph/embeddings.json): model + dim + vectors + the persisted cluster partition. Reclustering runs at the END of every embedding batch (post-merge hook, both front doors) — never per read (the agglomerative pass is ~1.5s at 338 nodes). - Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers stay singletons by default; attachStragglers is the all-in toggle. Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans land in real topics, 10 stragglers. - Consumers (snapshot, settle, namer, replay) read the persisted partition — one cluster concept everywhere; core-survival label matching unchanged (20/26 topics kept labels). - Louvain retired: detectCommunities + graphology* deps deleted (graphology-louvain had leaked into the client bundle since the clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed. - scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s including the one-time model download).
This commit is contained in:
@@ -1,5 +1,4 @@
|
||||
import type { GraphEdge, GraphNode } from "./contracts.ts";
|
||||
import { detectCommunities } from "./clusters.ts";
|
||||
import type { EmbeddingsLedger, GraphEdge, GraphNode } from "./contracts.ts";
|
||||
import { incrementalLayout } from "./layout.ts";
|
||||
import type { Position } from "./contracts.ts";
|
||||
|
||||
@@ -13,7 +12,7 @@ import type { Position } from "./contracts.ts";
|
||||
*
|
||||
* SPLIT FROM layout.ts deliberately (2026-09-13): layout.ts is in the
|
||||
* browser barrel, and the settle needs the Louvain partition for the
|
||||
* cluster centroid pull — graphology-louvain must not ship in the client
|
||||
* cluster centroid pull — the heavy deps must not ship in the client
|
||||
* bundle. Server code imports this module DIRECTLY; it is never
|
||||
* re-exported through src/core/index.ts.
|
||||
*/
|
||||
@@ -22,6 +21,7 @@ export function settleGraphLayout(graphStore: {
|
||||
readEdges(): GraphEdge[];
|
||||
readPositions(): Record<string, Position>;
|
||||
writePositions(positions: Record<string, Position>): void;
|
||||
readEmbeddings(): EmbeddingsLedger;
|
||||
}): void {
|
||||
const nodes = graphStore.readNodes();
|
||||
if (nodes.length === 0) return;
|
||||
@@ -34,10 +34,11 @@ export function settleGraphLayout(graphStore: {
|
||||
});
|
||||
const edges = graphStore.readEdges();
|
||||
const previous = new Map(Object.entries(graphStore.readPositions()));
|
||||
// The cluster partition for the centroid pull: same Louvain the
|
||||
// snapshot uses (same nodes/edges = same communities). 338 nodes:
|
||||
// ~1-2ms, noise next to the settle itself.
|
||||
const { communities } = detectCommunities(unique, edges);
|
||||
// The cluster partition for the centroid pull: the SAME persisted
|
||||
// semantic topics the snapshot renders (computed by the embedder
|
||||
// once per batch, stored in the embedding ledger) — one cluster
|
||||
// concept everywhere.
|
||||
const communities = graphStore.readEmbeddings().clusters;
|
||||
const next = incrementalLayout(unique, edges, previous, { clusters: communities });
|
||||
graphStore.writePositions(Object.fromEntries(next));
|
||||
}
|
||||
Reference in New Issue
Block a user