sorokin.oleg.edu/learnharness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Semantic topic clusters: local embeddings replace Louvain

Browse Source 
Louvain only ever found the reasoning chains (the graph is disconnected
by nature — every connected component became its own cluster) and
edge-orphans could never join anything. Clusters are now RELATED TOPICS:
propositions grouped by meaning.

- Embedder (src/server/embeddings.ts): in-process transformers.js v4
  running EmbeddingGemma-300M q8 (MTEB 68.4, ~300MB one-time download),
  MRL-truncated to 512d, "title: none | text:" prefix on 100% of inputs,
  L2-normalized so cosine = dot. Model/dim mismatch voids the ledger
  (full re-embed). Injected at the composition root — the agent layer
  never imports the server layer (architecture law).
- Ledger (data/graph/embeddings.json): model + dim + vectors + the
  persisted cluster partition. Reclustering runs at the END of every
  embedding batch (post-merge hook, both front doors) — never per read
  (the agglomerative pass is ~1.5s at 338 nodes).
- Clustering (src/core/semantic-clusters.ts): hand-rolled average-linkage
  agglomerative over cosine (npm HDBSCAN is an abandoned alpha). Stragglers
  stay singletons by default; attachStragglers is the all-in toggle.
  Threshold 0.60 swept on the real graph: 26 topics, 74/90 edge-orphans
  land in real topics, 10 stragglers.
- Consumers (snapshot, settle, namer, replay) read the persisted
  partition — one cluster concept everywhere; core-survival label
  matching unchanged (20/26 topics kept labels).
- Louvain retired: detectCommunities + graphology* deps deleted
  (graphology-louvain had leaked into the client bundle since the
  clusters wave); scripts/bench-graph.ts (dead FA2 bench) removed.
- scripts/embed-graph.ts: backfill/cold-start (338 nodes in 150s
  including the one-time model download).
This commit is contained in:
amosblomqvist
2026-09-13 14:29:16 +03:00
parent 3a1c18b9de
commit f5edc7f0a5
28 changed files with 1443 additions and 431 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/core/clusters.ts
+12 -108
View File
@@ -1,31 +1,19 @@
/**
* COMMUNITY DETECTION for the knowledge graph (vision §1, DECIDED
* 2026-09-13): Louvain modularity over the undirected, unweighted
* follows+supports structure — purely structural, so it never depends on
* layout, and the extraction pipeline never knows clusters exist.
* CLUSTER LABEL MATCHING (vision §1). DETECTION moved to semantic topic
* clustering over embeddings (core/semantic-clusters.ts, DECIDED
* 2026-09-13) — Louvain over the edge structure is retired: it only ever
* found the reasoning chains (the graph is disconnected by nature), and
* edge-orphans could never join anything. What REMAINS here is the
* cluster contract + core-survival label matching, which is
* detection-agnostic: it consumes the same { members } shape semantic
* clustering emits.
*
* Determinism: Louvain's random-walk traversal is driven by a SEEDED
* mulberry32 RNG (default seed constant below), so the same graph always
* yields the same partition. Community INDICES are still not stable
* across graph edits — which is why labelings (Phase B) key to sorted
* MEMBER SETS, never to indices.
* Labelings key to sorted MEMBER SETS, never to detection indices —
* a labeling record outlives any reshuffle of the partition.
*
* Undirected + unweighted (user-decided 2026-09-13): both edge types are
* equally real relations between propositions — direction is
* derivational-vs-grounding, not stronger-vs-weaker — and a follows AND
* a supports between the same pair collapse to one edge.
*
* This module is browser-safe pure logic (no node builtins, no fs): the
* server runs it for the snapshot payload, the renderer for nothing
* (clusters arrive precomputed), and tests run it directly.
*
* NOTE: no minimum-size filter here — detection returns every community,
* singletons included. The ≥3-node render threshold (decided) is a
* PRESENTATION rule; it lives in the payload derivation and the
* renderer, not in the algorithm.
* Browser-safe pure logic (no node builtins, no fs): the server runs
* matching for the snapshot payload, tests run it directly.
*/
import Graph from "graphology";
import louvain from "graphology-communities-louvain";
import type { ClusterLabeling } from "./contracts.ts";
/** A detected community. Members are SORTED node ids — the member set IS
@@ -39,90 +27,6 @@ export interface Community {
* least three propositions to be worth a hull and a label. */
export const MIN_CLUSTER_SIZE = 3;
export interface CommunityDetection {
communities: Community[];
/** Modularity of the found partition — the one quality number. */
modularity: number;
}
export interface DetectOptions {
/** Louvain resolution (higher → more, smaller communities). Default 1.
* Parked for the multi-scale work; exposed now so that dial has a home. */
resolution?: number;
/** RNG seed. Default "lh-clusters". Same seed + same graph → same
* partition, always. */
seed?: string;
}
/** mulberry32 seeded by a string (FNV-1a) — the same recipe as
* question-core's seededRandom, inlined so this module has no
* cross-domain dependency. */
function seededRng(seed: string): () => number {
let h = 2166136261;
for (let i = 0; i < seed.length; i++) {
h ^= seed.charCodeAt(i);
h = Math.imul(h, 16777619);
}
let a = h >>> 0;
return () => {
a |= 0;
a = (a + 0x6d2b79f5) | 0;
let t = Math.imul(a ^ (a >>> 15), 1 | a);
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
/**
* Detect communities over a graph's structure. Inputs are the minimal
* structural shapes of the snapshot contracts (GraphNode.id,
* GraphEdge.from/to), so both the real snapshot and test fixtures feed it.
*
* Edges referencing unknown node ids are skipped defensively (the ledgers
* are append-only; a mid-merge read should never crash detection), and
* duplicate pairs (follows+supports, or reverse direction) collapse —
* mergeEdge keeps the first.
*/
export function detectCommunities(
nodes: readonly { id: string }[],
edges: readonly { from: string; to: string }[],
opts?: DetectOptions,
): CommunityDetection {
const graph = new Graph({ type: "undirected" });
for (const node of nodes) graph.addNode(node.id);
for (const edge of edges) {
if (!graph.hasNode(edge.from) || !graph.hasNode(edge.to)) continue;
if (edge.from === edge.to) continue; // self-loops: no community signal
graph.mergeEdge(edge.from, edge.to);
}
// Louvain on a 0-edge graph is degenerate; every node is trivially its
// own community (and louvain on an empty graph returns nothing useful).
if (graph.size === 0) {
return {
communities: nodes.map((n) => ({ members: [n.id] })).sort((a, b) => (a.members[0] ?? "").localeCompare(b.members[0] ?? "")),
modularity: 0,
};
}
const details = louvain.detailed(graph, {
rng: seededRng(opts?.seed ?? "lh-clusters"),
resolution: opts?.resolution ?? 1,
});
const byIndex = new Map<number, string[]>();
for (const [id, index] of Object.entries(details.communities)) {
const bucket = byIndex.get(index) ?? [];
bucket.push(id);
byIndex.set(index, bucket);
}
const communities = [...byIndex.values()]
.map((ids) => ({ members: ids.sort() }))
.sort(
(a, b) =>
b.members.length - a.members.length ||
(a.members[0] ?? "").localeCompare(b.members[0] ?? ""),
);
return { communities, modularity: details.modularity };
}
/**
* Claim labeling records for detected communities — CORE-SURVIVAL (DECIDED
* 2026-09-13): a community keeps a record's label while AT LEAST HALF of