Semantic topic clusters: local embeddings replace Louvain
Louvain only ever found the reasoning chains (the graph is disconnected by nature — every connected component became its own cluster) and edge-orphans could never join anything. Clusters are now RELATED TOPICS: propositions grouped by meaning. - Embedder (src/server/embeddings.ts): in-process transformers.js v4 running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download), MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs, L2-normalized so cosine = dot. Model/dim mismatch voids the ledger (full re-embed). Injected at the composition root — the agent layer never imports the server layer (architecture law). - Ledger (data/graph/embeddings.json): model + dim + vectors + the persisted cluster partition. Reclustering runs at the END of every embedding batch (post-merge hook, both front doors) — never per read (the agglomerative pass is ~1.5s at 338 nodes). - Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers stay singletons by default; attachStragglers is the all-in toggle. Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans land in real topics, 10 stragglers. - Consumers (snapshot, settle, namer, replay) read the persisted partition — one cluster concept everywhere; core-survival label matching unchanged (20/26 topics kept labels). - Louvain retired: detectCommunities + graphology* deps deleted (graphology-louvain had leaked into the client bundle since the clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed. - scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s including the one-time model download).
This commit is contained in:
+13
-9
@@ -51,7 +51,7 @@ import type {
|
||||
LedgerEvent,
|
||||
LedgerEventKind,
|
||||
} from "./contracts.ts";
|
||||
import { detectCommunities, matchClusterLabelings, MIN_CLUSTER_SIZE } from "./clusters.ts";
|
||||
import { matchClusterLabelings, MIN_CLUSTER_SIZE } from "./clusters.ts";
|
||||
import type { QuestionAnswerData } from "./question-core.ts";
|
||||
import type { ChatStore, GraphStore } from "./ports.ts";
|
||||
|
||||
@@ -497,14 +497,18 @@ export function graphSnapshot(deps: GraphDeps): GraphSnapshot {
|
||||
(nodeObservations[nodeId] ??= {})[chatId] = note;
|
||||
}
|
||||
|
||||
// Structural communities (vision §1): Louvain over the undirected
|
||||
// follows+supports structure, joined with persisted labelings —
|
||||
// core-survival matching, size-filtered at the presentation threshold.
|
||||
// Detection is DETERMINISTIC (seeded), so no memo in v1: a snapshot
|
||||
// read recomputes it (ms at current scale; revisit at vault scale).
|
||||
const detection = detectCommunities(nodes, edges);
|
||||
const claims = matchClusterLabelings(detection.communities, deps.graphStore.readClusterLabels());
|
||||
const clusters: Cluster[] = detection.communities
|
||||
// Semantic topic communities (DECIDED 2026-09-13, replacing structural
|
||||
// Louvain): average-linkage clustering over embeddings — related
|
||||
// topics, not reasoning chains. Edge-orphans join topics by meaning;
|
||||
// stragglers stay loose (filtered at the presentation threshold
|
||||
// below). The partition is computed ONCE per embedding batch by the
|
||||
// embedder (the agglomerative pass is ~1.5s at current scale) and
|
||||
// PERSISTED in the embedding ledger — reads just load it. Nodes whose
|
||||
// embedding hasn't landed yet simply sit unclustered until the next
|
||||
// read after the embedder runs.
|
||||
const communities = deps.graphStore.readEmbeddings().clusters;
|
||||
const claims = matchClusterLabelings(communities, deps.graphStore.readClusterLabels());
|
||||
const clusters: Cluster[] = communities
|
||||
.map((community, index) => ({ community, claim: claims[index] }))
|
||||
.filter(({ community }) => community.members.length >= MIN_CLUSTER_SIZE)
|
||||
.map(({ community, claim }) => ({
|
||||
|
||||
Reference in New Issue
Block a user