Compare commits

...
16 changed files with 4483 additions and 189 deletions
+10 -10
View File
@@ -35,26 +35,26 @@ Use Gitea for live coordination and the relevant code version for implementation
## Coordination and integration
All independent agents coordinating through Gitea run on completely separate machines, each with its own clone, filesystem, tools, processes, and runtime. Gitea and pushed Git history are the shared communication and delivery channels. Local files (including `harness`), unpushed commits, worktrees, test results, and localhost services are not accessible to peers. Matching branch names or paths do not imply synchronized contents.
1. On starting or resuming, including after compaction, inspect the working branch, local changes, applicable instructions, relevant shared records, and existing implementation. Recover context from authoritative records. Check for newer shared instructions on the actual remote default branch without overwriting local work. Read what is relevant; avoid repeated full-history scans and constant polling.
2. Search before creating tasks. Claim a meaningful outcome in its Gitea issue with a stable session identity, scope, branch/worktree, and acceptance criteria. Keep small steps within that task. Recheck after claiming; comments are not atomic locks. Respect existing ownership and resolve overlapping claims before overlapping work.
3. Independent sessions are peers. Each owns its contribution and delegated work. Record one integration/release owner when a milestone needs shared coordination. Use separate worktrees or checkouts, partition files/interfaces, and coordinate changes to shared specifications and documents.
4. At meaningful checkpoints or handoffs, record changed results, commit/PR links, verification commands and environment, blockers, and next actions. Identify the tested code version and uncommitted or unpublished work; make required artifacts accessible to other sessions. Do not assume another agent is running or has read a message; silence alone does not establish abandonment. Record evidence and preserve existing work when reassigning a task.
5. Follow project review and merge conventions. Inspect upstream changes, resolve conflicts while preserving others' edits, and verify the combined result. Do not overwrite others' work or force-push shared history to resolve coordination problems.
2. Search before creating tasks. Claim a meaningful outcome in its Gitea issue with a stable session identity, scope, remote repository/branch, and acceptance criteria. Keep small steps within that task. Recheck after claiming; comments are not atomic locks. Respect existing ownership and resolve overlapping claims before overlapping work.
3. Independent sessions are peers. Each owns its contribution and delegated work on its own machine. Record one integration/release owner when a milestone needs shared coordination. Use distinct remote work branches, partition files/interfaces, and coordinate changes to shared specifications and documents. Worktrees provide local isolation only; they do not share work between machines.
4. At meaningful checkpoints or handoffs, record changed results, commit/PR links, verification commands and environment, blockers, and next actions. Identify the tested code version and label uncommitted or unpublished work as local-only. Before asking peers to consume results, push the required code and versioned documentation, including relevant `harness` updates; publish other required artifacts in Gitea and link them. Use repository-relative paths with commit references, reproducible commands, and necessary environment details; local paths and localhost URLs alone are not handoffs. Do not assume another agent is running or has read a message; silence alone does not establish abandonment. Record evidence and preserve existing work when reassigning a task.
5. Follow project review and merge conventions. Fetch the relevant remote branches and inspect the identified commits before reviewing or integrating another machine's work. Resolve conflicts while preserving others' edits, and verify the combined result locally; report material environment differences. Local instruction-file edits reach peers only after publication and retrieval. Do not overwrite others' work or force-push shared history to resolve coordination problems.
6. Close work when its acceptance criteria are met and the result is integrated or delivered as appropriate. Mark duplicates and superseded tasks explicitly with replacement links. Then choose further useful unowned work while the requested objective remains incomplete.
Use Gitea MCP for shared coordination. If unavailable, report the failure, buffer necessary handoff information locally, and continue already-owned or clearly non-overlapping work. Reconcile with current Gitea state when access returns; do not claim an ownership update, review, or publication succeeded without confirmation.
## Delegation and reasoning
Use GPT-6 Astra exclusively. Delegate bounded independent research, design, implementation, or review when it improves progress, judgment, or preserves the parent's context. Give subagents the relevant context, ownership boundaries, result format, and evidence requirements. Use fresh critics when practical. The parent owns synthesis and integration and normally consolidates results into Gitea. Choose concurrency according to useful independent work and available resources, including other sessions; there is no agent-count quota.
Use GPT-6 Astra exclusively. Gitea peers are independent remote sessions, not subagents controlled by this session. Spawned subagents use the execution environment actually provided by the agent tools; do not assume they share files, and publish their results before Gitea peers need them. Delegate bounded independent research, design, implementation, or review when it improves progress, judgment, or preserves the parent's context. Give subagents the relevant context, ownership boundaries, result format, and evidence requirements. Use fresh critics when practical. The parent owns synthesis and integration and normally consolidates results into Gitea. Choose concurrency according to useful independent work and available resources, including other sessions; there is no agent-count quota.
Optimize reasoning effort for verified outcomes per cost, including review, retries, and repair:
Main agent: default to high for planning, synthesis, integration, and acceptance decisions; medium for dispatching a settled plan and routine coordination.
Subagents: default to medium for bounded implementation and targeted research; low for straightforward retrieval, mechanical edits, and running established checks; high for complex debugging, correctness review, adversarial analysis, or ambiguous design.
Use extra-high selectively for difficult foundational semantic, methodological, formal, or cross-domain decisions. Reserve max for exceptional unresolved problems where additional reasoning is likely to justify its cost. Retain higher settings when evidence supports their value.
- Main agent: default to high for planning, synthesis, integration, and acceptance decisions; medium for dispatching a settled plan and routine coordination.
- Subagents: default to medium for bounded implementation and targeted research; low for straightforward retrieval, mechanical edits, and running established checks; high for complex debugging, correctness review, adversarial analysis, or ambiguous design.
- Use extra-high selectively for difficult foundational semantic, methodological, formal, or cross-domain decisions. Reserve max for exceptional unresolved problems where additional reasoning is likely to justify its cost. Retain higher settings when evidence supports their value.
Allocate by task difficulty and consequences, not role or length alone. Start higher when a clearly difficult decision has expensive downstream consequences. Escalate for concrete reasoning difficulties; resolve missing evidence or unclear requirements at their source. Set subagent effort explicitly through supported controls to avoid unintended inheritance. Do not claim an unapplied change or alter global configuration merely to enforce these preferences.
+171 -28
View File
@@ -1,8 +1,8 @@
//! Typed OA/SA language, tolerant source analysis, and explicit V1/V2/V3/V4 migration.
mod raw_tokens;
#[path = "language/semantic_source.rs"]
mod semantic_source;
mod source3_bridge;
mod raw_tokens;
// Staged syntax foundation only. Public source8 admission waits for complete
// declaration lowering, source updates and the root-owned resource gate.
#[cfg(test)]
@@ -480,6 +480,8 @@ fn parse_document(source: &str, limits: ParseLimits) -> Result<ParsedDocument, V
cursor: 0,
value_depth: 0,
limits,
#[cfg(test)]
mixed: None,
}
.document()
.map_err(|e| vec![diagnostic(source, e)])
@@ -914,6 +916,8 @@ pub(crate) fn parse_unit(
cursor: 0,
value_depth: 0,
limits,
#[cfg(test)]
mixed: None,
}
.unit(entry, true)
.map_err(|e| Box::new(diagnostic(source, e)))
@@ -1005,31 +1009,52 @@ fn edit_positions(
unit: SourceUnit,
) -> Result<Vec<(Id, EditDeclaration)>, Box<Diagnostic>> {
let tokens = strict_tokens(source).map_err(|e| Box::new(diagnostic(source, e)))?;
let mut ends: Vec<_> = tokens.iter().map(|token| token.span.end).collect();
edit_positions_from_tokens(source, &unit, |at| tokens[at], tokens.len(), None)
}
fn edit_positions_from_tokens(
source: &str,
unit: &SourceUnit,
token: impl Fn(usize) -> Lexeme,
count: usize,
value_ends: Option<&BTreeMap<usize, usize>>,
) -> Result<Vec<(Id, EditDeclaration)>, Box<Diagnostic>> {
let mut ends: Vec<_> = (0..count).map(|at| token(at).span.end).collect();
let mut stack = Vec::new();
for (index, token) in tokens.iter().enumerate() {
for index in 0..count {
let token = token(index);
match token.text(source) {
"{" | "[" => stack.push(index),
"}" | "]" => ends[stack.pop().expect("parser checked delimiters")] = token.span.end,
_ => {}
}
}
struct Positions<'a> {
tokens: &'a [Lexeme],
struct Positions<'a, F> {
token_at: F,
ends: Vec<usize>,
value_ends: Option<&'a BTreeMap<usize, usize>>,
}
impl Positions<'_> {
impl<F: Fn(usize) -> Lexeme> Positions<'_, F> {
fn token(&self, start: usize) -> usize {
self.tokens
.partition_point(|token| token.span.start < start)
let mut low = 0;
let mut high = self.ends.len();
while low < high {
let middle = low + (high - low) / 2;
if (self.token_at)(middle).span.start < start {
low = middle + 1;
} else {
high = middle;
}
}
low
}
fn fields(&self, fields: &BTreeMap<String, Ast>) -> BTreeMap<String, EditField> {
fields
.iter()
.map(|(key, ast)| {
let value = self.value(ast);
let start = self.tokens[self.token(value.span.start) - 1].span.start;
let end = self.tokens[self.token(value.span.end)].span.end;
let start = (self.token_at)(self.token(value.span.start) - 1).span.start;
let end = (self.token_at)(self.token(value.span.end)).span.end;
(
key.clone(),
EditField {
@@ -1044,10 +1069,15 @@ fn edit_positions(
let index = self.token(ast.offset());
let span = Span {
start: ast.offset(),
end: match ast {
Ast::Reference(..) | Ast::HistoricalUuid(..) => self.tokens[index + 1].span.end,
_ => self.ends[index],
},
end: self
.value_ends
.and_then(|ends| ends.get(&ast.offset()).copied())
.unwrap_or_else(|| match ast {
Ast::Reference(..) | Ast::HistoricalUuid(..) => {
(self.token_at)(index + 1).span.end
}
_ => self.ends[index],
}),
};
let kind = match ast {
Ast::Object(fields, _) => EditKind::Record(self.fields(fields)),
@@ -1062,11 +1092,12 @@ fn edit_positions(
}
}
let positions = Positions {
tokens: &tokens,
token_at: &token,
ends,
value_ends,
};
unit.declarations
.into_iter()
.iter()
.map(|declaration| {
if declaration.id_insertion.is_some() {
return Err(Box::new(diagnostic(
@@ -1086,8 +1117,8 @@ fn edit_positions(
let start = positions.token(declaration.span.start);
let header = roles
.iter()
.zip(&tokens[start..])
.map(|(role, token)| (*role, token.span))
.zip(start..count)
.map(|(role, at)| (*role, token(at).span))
.collect();
Ok((
declaration.id,
@@ -1114,10 +1145,35 @@ struct Parser<'a> {
cursor: usize,
value_depth: usize,
limits: ParseLimits,
#[cfg(test)]
mixed: Option<analysis_source::mixed::State<'a>>,
}
impl Parser<'_> {
fn token(&self, index: usize) -> Option<Lexeme> {
#[cfg(test)]
if let Some(mixed) = &self.mixed {
return mixed.token(index);
}
self.tokens.get(index).copied()
}
fn current(&self) -> Lexeme {
self.tokens[self.cursor.min(self.tokens.len() - 1)]
self.token(self.cursor).unwrap_or(Lexeme {
class: TokenClass::Punctuation,
span: Span {
start: self.source.len(),
end: self.source.len(),
},
})
}
fn syntax_entry(&mut self) -> Result<(), Error> {
#[cfg(test)]
{
let span = self.current().span;
if let Some(mixed) = &mut self.mixed {
mixed.entry(span)?;
}
}
Ok(())
}
fn text(&self) -> &str {
self.current().text(self.source)
@@ -1154,6 +1210,10 @@ impl Parser<'_> {
}
fn optional_id(&mut self) -> Result<Id, Error> {
if self.text() != "id" {
#[cfg(test)]
if self.mixed.is_some() {
return Err(self.error("Private mixed syntax requires explicit IDs."));
}
return Ok(Id::new_v4());
}
self.cursor += 1;
@@ -1194,6 +1254,7 @@ impl Parser<'_> {
Ok(fields)
}
fn field(&mut self) -> Result<(String, Ast, Span), Error> {
self.syntax_entry()?;
let key = self.text().to_owned();
if !matches!(
self.current().class,
@@ -1204,15 +1265,19 @@ impl Parser<'_> {
self.cursor += 1;
let start = self.current().span.start;
let value = self.value()?;
let end = self.tokens[self.cursor - 1].span.end;
let end = self
.token(self.cursor - 1)
.expect("consumed token")
.span
.end;
self.take(";")?;
Ok((key, value, Span { start, end }))
}
fn nested_header_kind(&self) -> Option<DeclKind> {
let kind = declaration_kind(self.text())?;
let alias = self.tokens.get(self.cursor + 1)?;
let following = self.tokens.get(self.cursor + 2)?;
let alias = self.token(self.cursor + 1)?;
let following = self.token(self.cursor + 2)?;
// A node's ordinary `diagram @view;` field is not a declaration.
if !crate::valid_alias(alias.text(self.source)) {
return None;
@@ -1227,14 +1292,33 @@ impl Parser<'_> {
if self.value_depth >= 64 {
return Err(self.error("Typed values may not nest more than 64 levels."));
}
self.syntax_entry()?;
self.value_depth += 1;
let result = self.value_inner();
self.value_depth -= 1;
#[cfg(test)]
if let Ok(ast) = &result {
let end = self
.token(self.cursor - 1)
.expect("consumed value")
.span
.end;
if let Some(mixed) = &mut self.mixed {
mixed.value_ends.insert(ast.offset(), end);
}
}
result
}
fn value_inner(&mut self) -> Result<Ast, Error> {
let token = self.current();
let offset = token.span.start;
#[cfg(test)]
if let Some(mixed) = &self.mixed
&& (token.class == TokenClass::Number || matches!(self.text(), "-" | "+"))
{
let value = mixed.ordinary_number(&mut self.cursor)?;
return Ok(Ast::Literal(value, offset));
}
match self.text() {
"uuid" => {
self.cursor += 1;
@@ -1332,16 +1416,31 @@ impl Parser<'_> {
fn unit(&mut self, entry: bool, includes_allowed: bool) -> Result<SourceUnit, Error> {
let mut header = if entry {
self.take("arcadia")?;
if !matches!(self.text(), "3" | "4" | "5" | "6" | "7") {
return Err(
self.error("Unsupported schema version: expected arcadia 3, 4, 5, 6 or 7.")
);
#[cfg(test)]
let mixed_mode = self.mixed.is_some();
#[cfg(not(test))]
let mixed_mode = false;
if if mixed_mode {
self.text() != "8"
} else {
!matches!(self.text(), "3" | "4" | "5" | "6" | "7")
} {
return Err(self.error(if mixed_mode {
"Private mixed syntax requires arcadia 8."
} else {
"Unsupported schema version: expected arcadia 3, 4, 5, 6 or 7."
}));
}
let schema_version = self.text().parse().expect("checked source version");
self.cursor += 1;
self.take(";")?;
let start = self.current().span.start;
self.take("project")?;
#[cfg(test)]
if let Some(mixed) = &mut self.mixed {
mixed.record(false, Span { start, end: start })?;
}
self.syntax_entry()?;
let alias = self.identifier()?;
let name = self.quoted()?;
let id_insertion = (self.text() != "id").then_some(self.current().span.start);
@@ -1382,6 +1481,14 @@ impl Parser<'_> {
let mut open: Vec<usize> = Vec::new();
loop {
let offset = self.current().span.start;
#[cfg(test)]
if let Some(mixed) = &mut self.mixed
&& mixed.typed_header(self.cursor, open.is_empty())
{
let owner = open.last().map(|&at| declarations[at].id);
mixed.parse_typed(&mut self.cursor, owner)?;
continue;
}
if let Some(&owner) = open.last() {
if self.text() == "}" {
declarations[owner].body.end = offset;
@@ -1425,6 +1532,14 @@ impl Parser<'_> {
if !includes_allowed {
return Err(self.error("An include requires a source bundle; single-string parsing never reads files."));
}
#[cfg(test)]
if let Some(mixed) = &mut self.mixed {
mixed.include(Span {
start: offset,
end: offset,
})?;
}
self.syntax_entry()?;
self.cursor += 1;
let path = self.quoted()?;
self.take(";")?;
@@ -1432,7 +1547,11 @@ impl Parser<'_> {
path,
Span {
start: offset,
end: self.tokens[self.cursor - 1].span.end,
end: self
.token(self.cursor - 1)
.expect("consumed token")
.span
.end,
},
));
continue;
@@ -1444,20 +1563,40 @@ impl Parser<'_> {
if entry_header.description.is_some() {
return Err(self.error("Duplicate project description."));
}
self.syntax_entry()?;
self.cursor += 1;
let description = self.quoted()?;
self.take(";")?;
entry_header.project.description = description;
entry_header.description = Some(Span {
start: offset,
end: self.tokens[self.cursor - 1].span.end,
end: self
.token(self.cursor - 1)
.expect("consumed token")
.span
.end,
});
header = Some(entry_header);
continue;
}
#[cfg(test)]
if self.mixed.is_some() && analysis_source::mixed::long_analysis_word(self.text()) {
return Err(self.error("Use the short source8 analysis declaration word."));
}
self.syntax_entry()?;
let word = self.text().to_owned();
let kind = declaration_kind(&word)
.ok_or_else(|| self.error(format!("Unknown declaration '{word}'.")))?;
#[cfg(test)]
if let Some(mixed) = &mut self.mixed {
mixed.record(
matches!(kind, DeclKind::Element(..) | DeclKind::Relation(..)),
Span {
start: offset,
end: offset,
},
)?;
}
graph_count += usize::from(matches!(
kind,
DeclKind::Element(..) | DeclKind::Relation(..)
@@ -1485,6 +1624,10 @@ impl Parser<'_> {
let body_start = self.current().span.end;
self.take("{")?;
let lexical_owner = open.last().map(|&owner| declarations[owner].id);
#[cfg(test)]
if let Some(mixed) = &mut self.mixed {
mixed.lexical_owners.push(open.last().copied());
}
declarations.push(Declaration {
kind,
alias,
@@ -3,9 +3,12 @@
mod budget;
mod cursor;
mod declarations;
mod index;
mod lex;
mod lower;
pub(super) mod mixed;
mod syntax;
mod update;
mod write;
#[cfg(test)]
@@ -143,6 +143,14 @@ impl Budget {
}
Ok(())
}
/// Aggregate owned text for an index/update phase, checked before copying.
pub fn index_bytes(&self, count: usize, span: Span) -> Result<(), Error> {
self.active()?;
if count > self.limits.bytes {
return Err(error(span, "Source8 index/output byte limit exceeded."));
}
Ok(())
}
pub fn lowering_occurrences(&self, count: usize, span: Span) -> Result<(), Error> {
self.active()?;
// Parameter declarations are syntax entries, not reference uses.
@@ -0,0 +1,709 @@
//! Temporary typed source witnesses. No arena index becomes a model identity.
//! The staged entry point parses each supplied typed file once. A future bundle
//! parser must retain the same opaque parse/lower provenance. Occurrence
//! completeness trusts that lowerer; the private builder does not prove the
//! presence of every syntax reference in a caller-fabricated Lowered value.
use super::cursor::Cursor;
use super::lower::{self, Child, Field, Lowered, Role, Target, Unit};
use super::{Budget, Error, Span, declarations, error, lex};
use crate::{Id, Project, analysis as a};
use std::cmp::Ordering;
use std::collections::{BTreeMap, BTreeSet};
use std::fmt;
use std::sync::Arc;
/// Persistent structural path: cloning shares ancestors, and equality/order
/// compare the complete edge sequence, never allocation or arena identities.
/// Depth is checked against the inherited frame limit before allocation.
/// Comparison and Debug walk leaf-to-root without allocating or recursing;
/// this private total order need not be root-first lexicographic order.
#[derive(Clone, Default)]
pub(super) struct Path(Option<Arc<Step>>);
struct Step {
parent: Path,
edge: Child,
}
impl PartialEq for Path {
fn eq(&self, other: &Self) -> bool {
self.cmp(other) == Ordering::Equal
}
}
impl Eq for Path {}
impl PartialOrd for Path {
fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
Some(self.cmp(other))
}
}
impl Ord for Path {
fn cmp(&self, other: &Self) -> Ordering {
let (mut left, mut right) = (self, other);
loop {
match (left.0.as_deref(), right.0.as_deref()) {
(Some(a), Some(b)) => {
let order = a.edge.cmp(&b.edge);
if order != Ordering::Equal {
return order;
}
(left, right) = (&a.parent, &b.parent);
}
(None, None) => return Ordering::Equal,
(None, Some(_)) => return Ordering::Less,
(Some(_), None) => return Ordering::Greater,
}
}
}
}
impl fmt::Debug for Path {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
// Debug intentionally shows the same leaf-to-root order as comparison.
let mut list = formatter.debug_list();
let mut path = self;
while let Some(step) = &path.0 {
list.entry(&step.edge);
path = &step.parent;
}
list.finish()
}
}
impl Drop for Path {
fn drop(&mut self) {
let mut next = self.0.take();
while let Some(link) = next {
// Shared ancestors remain owned elsewhere. The final owner peels
// unique links here, taking the parent before Step is dropped.
let Some(mut step) = Arc::into_inner(link) else {
break;
};
next = step.parent.0.take();
}
}
}
impl Path {
pub fn edges(&self) -> Vec<Child> {
let mut result = Vec::new();
let mut path = self;
while let Some(step) = &path.0 {
result.push(step.edge);
path = &step.parent;
}
result.reverse();
result
}
}
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub(super) struct ExpressionKey {
pub owner: Id,
pub field: Field,
pub path: Path,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub(super) enum TokenRole {
Declaration,
Use(Role),
}
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub(super) struct OccurrenceKey {
pub owner: Id,
/// None is a global declaration header, not a binding field.
pub field: Option<Field>,
/// None is outside an expression; Some(empty) is its root call site.
pub path: Option<Path>,
pub role: TokenRole,
pub target: Target,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(super) struct Location {
pub file: usize,
pub span: Span,
}
#[derive(Debug)]
pub(super) struct Index<'s> {
// Private provenance: callers can inspect locations, not supply replacements
// for the source witness, keys, catalog or occurrence spans.
sources: &'s BTreeMap<usize, String>,
names: BTreeMap<Target, String>,
occurrences: BTreeMap<OccurrenceKey, Location>,
expressions: BTreeMap<ExpressionKey, Location>,
}
impl<'s> Index<'s> {
pub fn sources(&self) -> &BTreeMap<usize, String> {
self.sources
}
pub fn names(&self) -> &BTreeMap<Target, String> {
&self.names
}
pub fn occurrences(&self) -> &BTreeMap<OccurrenceKey, Location> {
&self.occurrences
}
pub fn expressions(&self) -> &BTreeMap<ExpressionKey, Location> {
&self.expressions
}
/// Typed-only staged adapter. `ordinary` has already been parsed by its
/// owner. Missing typed IDs fail in lowering; this never invents UUIDs.
/// The caller merges the returned additions and admits the complete model.
pub fn parse(
ordinary: &Project,
sources: &'s BTreeMap<usize, String>,
budget: &mut Budget,
) -> Result<(Self, Lowered), lower::Failure> {
let mut units = Vec::new();
let mut witnesses = BTreeMap::new();
for (&file, source) in sources {
let failure = |error| lower::Failure {
file: Some(file),
error,
};
budget
.lowering_entries(units.len() + 1, Span::default())
.map_err(failure)?;
let tokens = lex::tokens(source, budget).map_err(failure)?;
let mut cursor = Cursor::new(source, &tokens, budget);
let mut parsed = Vec::new();
while cursor.cursor < tokens.len() {
parsed.push(declarations::parse(&mut cursor, 8, None).map_err(failure)?);
}
units.push(Unit {
file,
declarations: parsed,
});
witnesses.insert(file, tokens);
}
let lowered = lower::lower(ordinary, &units, budget)?;
// A build failure after our own parse/lower is a phase/invariant error,
// not an invented file assignment for an ambiguous numeric span.
let index = Self::build(ordinary, sources, &units, &witnesses, &lowered, budget)
.map_err(|error| lower::Failure { file: None, error })?;
Ok((index, lowered))
}
// Same-parse/lower internal seam only. Checks below reject damaged supplied
// sites but occurrence completeness is an invariant of our own lowerer.
// Do not expose a constructor accepting independently fabricated Lowered.
fn build(
ordinary: &Project,
sources: &'s BTreeMap<usize, String>,
units: &[Unit],
witnesses: &BTreeMap<usize, Vec<lex::Token>>,
lowered: &Lowered,
budget: &mut Budget,
) -> Result<Self, Error> {
budget.active()?;
budget.lowering_entries(lowered.expressions.len(), Span::default())?;
budget.lowering_occurrences(lowered.occurrences.len(), Span::default())?;
let mut names = BTreeMap::new();
let mut name_bytes = 0usize;
let mut insert_name = |target, name: &str| -> Result<(), Error> {
budget.lowering_entries(names.len() + 1, Span::default())?;
name_bytes = name_bytes
.checked_add(name.len())
.ok_or_else(|| error(Span::default(), "Index name size overflow."))?;
budget.index_bytes(name_bytes, Span::default())?;
if names.insert(target, name.to_owned()).is_some() {
return Err(error(
Span::default(),
"Duplicate qualified catalog identity.",
));
}
Ok(())
};
// Lowering has already bounded and validated the complete ordinary
// identity catalog before this existing convenience visitor allocates.
for (id, alias) in ordinary.identities() {
insert_name(Target::Global(id), alias)?;
}
for definition in ordinary
.definitions
.values()
.chain(lowered.definitions.values())
{
if lowered.definitions.contains_key(&definition.id) {
insert_name(Target::Global(definition.id), &definition.alias)?;
}
if let a::DefinitionBody::Calculation(calculation) = &definition.body {
for parameter in &calculation.parameters {
insert_name(
Target::Parameter {
calculation: definition.id,
parameter: parameter.id,
},
&parameter.name,
)?;
}
}
}
for element in lowered.elements.values() {
insert_name(Target::Global(element.id), &element.alias)?;
}
let mut index = Self {
sources,
names,
occurrences: BTreeMap::new(),
expressions: BTreeMap::new(),
};
let mut owners = BTreeMap::new();
let mut physical = BTreeSet::new();
for unit in units {
for declaration in &unit.declarations {
let owner = declaration.header.identity.value.ok_or_else(|| {
error(
declaration.span,
"Missing ID requires source stabilization.",
)
})?;
let location = Location {
file: unit.file,
span: declaration.span,
};
checked_source(sources, location)?;
budget.lowering_entries(owners.len() + 1, declaration.span)?;
if owners.insert(owner, location).is_some() {
return Err(error(declaration.span, "Duplicate source owner."));
}
let key = OccurrenceKey {
owner,
field: None,
path: None,
role: TokenRole::Declaration,
target: Target::Global(owner),
};
index.insert(
key,
Location {
file: unit.file,
span: declaration.header.alias.span,
},
witnesses,
&mut physical,
budget,
)?;
}
}
// Parent-before-child is part of the lowering contract. One forward
// pass rejects forward, cyclic, cross-file/owner/field and duplicate
// paths. Each node allocates one shared edge, not a copied path vector.
let mut paths: Vec<(ExpressionKey, usize)> = Vec::new();
for (position, site) in lowered.expressions.iter().enumerate() {
if !matches!(
site.field,
Field::Body | Field::Base | Field::Predicate | Field::Override(_)
) {
return Err(error(
site.span,
"Expression occurs outside a binding field.",
));
}
let location = Location {
file: site.file,
span: site.span,
};
checked_source(sources, location)?;
let owner = owners
.get(&site.owner)
.ok_or_else(|| error(site.span, "Unknown expression owner."))?;
contained(*owner, location)?;
let (path, depth) = if let Some((parent, edge)) = site.parent {
if parent >= position {
return Err(error(
site.span,
"Expression parent must precede its child.",
));
}
let ancestor = &lowered.expressions[parent];
if ancestor.owner != site.owner
|| ancestor.field != site.field
|| ancestor.file != site.file
{
return Err(error(
site.span,
"Expression parent crosses its owner, field or file.",
));
}
contained(
Location {
file: ancestor.file,
span: ancestor.span,
},
location,
)?;
if ancestor.span == site.span {
return Err(error(
site.span,
"Child expression must be a proper source subtree.",
));
}
let depth = paths[parent].1 + 1;
budget.frame(depth, site.span)?;
(
Path(Some(Arc::new(Step {
parent: paths[parent].0.path.clone(),
edge,
}))),
depth,
)
} else {
budget.frame(1, site.span)?;
(Path::default(), 1)
};
let key = ExpressionKey {
owner: site.owner,
field: site.field,
path,
};
if index.expressions.insert(key.clone(), location).is_some() {
return Err(error(site.span, "Duplicate qualified expression path."));
}
paths.push((key, depth));
}
for occurrence in &lowered.occurrences {
let location = Location {
file: occurrence.file,
span: occurrence.span,
};
let owner = owners
.get(&occurrence.owner)
.ok_or_else(|| error(occurrence.span, "Unknown occurrence owner."))?;
contained(*owner, location)?;
let path = match occurrence.expression {
Some(site) => {
let expression = lowered
.expressions
.get(site)
.ok_or_else(|| error(occurrence.span, "Unknown occurrence expression."))?;
if expression.owner != occurrence.owner
|| expression.field != occurrence.field
|| expression.file != occurrence.file
{
return Err(error(
occurrence.span,
"Occurrence crosses its expression owner, field or file.",
));
}
contained(
Location {
file: expression.file,
span: expression.span,
},
location,
)?;
Some(paths[site].0.path.clone())
}
None => None,
};
match (occurrence.role, occurrence.target) {
(
Role::ParameterDeclaration,
Target::Parameter {
calculation,
parameter,
},
) if calculation == occurrence.owner
&& occurrence.field == Field::Input(parameter)
&& path.is_none() => {}
(Role::ParameterUse, Target::Parameter { calculation, .. })
if calculation == occurrence.owner
&& occurrence.field == Field::Body
&& path.is_some() => {}
(Role::ArgumentLabel, Target::Parameter { .. }) if path.is_some() => {}
(Role::OverrideTarget, Target::Global(id))
if occurrence.field == Field::Override(id) && path.is_none() => {}
(Role::CallTarget, Target::Global(_)) if path.is_some() => {}
(Role::Reference, Target::Global(_)) => {}
_ => return Err(error(occurrence.span, "Invalid qualified occurrence role.")),
}
let key = OccurrenceKey {
owner: occurrence.owner,
field: Some(occurrence.field),
path,
role: TokenRole::Use(occurrence.role),
target: occurrence.target,
};
index.insert(key, location, witnesses, &mut physical, budget)?;
}
// Argument labels belong to this precise call site and its callee, not
// merely to a parameter UUID/name present somewhere in the catalog.
let mut calls = BTreeMap::new();
for (key, at) in &index.occurrences {
if key.role == TokenRole::Use(Role::CallTarget) {
let site = (key.owner, key.field, key.path.clone());
if calls.insert(site, key.target).is_some() {
return Err(error(at.span, "Multiple callees at one expression site."));
}
}
}
for (key, at) in &index.occurrences {
if let (TokenRole::Use(Role::ArgumentLabel), Target::Parameter { calculation, .. }) =
(key.role, key.target)
&& calls.get(&(key.owner, key.field, key.path.clone()))
!= Some(&Target::Global(calculation))
{
return Err(error(
at.span,
"Argument label crosses its called calculation.",
));
}
}
for (position, site) in lowered.expressions.iter().enumerate() {
if let Some((parent, Child::Argument(parameter))) = site.parent {
let key = &paths[parent].0;
let Some(Target::Global(calculation)) =
calls.get(&(key.owner, Some(key.field), Some(key.path.clone())))
else {
return Err(error(site.span, "Argument child has no owning call."));
};
let label = OccurrenceKey {
owner: site.owner,
field: Some(site.field),
path: Some(key.path.clone()),
role: TokenRole::Use(Role::ArgumentLabel),
target: Target::Parameter {
calculation: *calculation,
parameter,
},
};
if !index.occurrences.contains_key(&label) || paths[position].0.path == key.path {
return Err(error(site.span, "Argument child has no qualified label."));
}
}
}
index.check_syntax_edges(units, lowered, budget)?;
Ok(index)
}
/// Check structural edges against the already parsed syntax, not just the
/// arena's internally consistent claims. This is one iterative tree walk,
/// with indexed span lookup; it does not lex or copy source/path strings.
fn check_syntax_edges<'a>(
&self,
units: &'a [Unit],
lowered: &Lowered,
budget: &Budget,
) -> Result<(), Error> {
let mut globals = BTreeMap::new();
let mut parameters = BTreeMap::new();
for (target, name) in &self.names {
match target {
Target::Global(id) => {
globals.insert(name.as_str(), *id);
}
Target::Parameter {
calculation,
parameter,
} => {
parameters.insert((*calculation, name.as_str()), *parameter);
}
}
}
let mut sites = BTreeMap::new();
for (position, site) in lowered.expressions.iter().enumerate() {
if sites
.insert(
(
site.file,
site.owner,
site.field,
site.span.start,
site.span.end,
),
position,
)
.is_some()
{
return Err(error(site.span, "Duplicate expression source ownership."));
}
}
let mut work = Vec::new();
let mut root = |file, owner, field, binding: &'a super::Binding| -> Result<(), Error> {
if let super::Binding::Expression(expression) = binding {
budget.lowering_entries(work.len() + 1, expression.span)?;
work.push((file, owner, field, expression, None));
}
Ok(())
};
for unit in units {
for declaration in &unit.declarations {
let owner = declaration.header.identity.value.expect("validated owner");
match &declaration.body {
declarations::Body::Calculation { body, .. } => {
root(unit.file, owner, Field::Body, &body.value)?
}
declarations::Body::Property { base, .. } => {
root(unit.file, owner, Field::Base, &base.value)?
}
declarations::Body::Check { predicate, .. } => {
root(unit.file, owner, Field::Predicate, &predicate.value)?
}
declarations::Body::Case { overrides, .. } => {
for entry in &overrides.value {
let target =
*globals.get(entry.target.name.as_str()).ok_or_else(|| {
error(entry.target.span, "Unknown syntax override target.")
})?;
root(unit.file, owner, Field::Override(target), &entry.binding)?;
}
}
_ => {}
}
}
}
let mut visited = 0usize;
while let Some((file, owner, field, expression, parent)) = work.pop() {
budget.lowering_entries(visited + 1, expression.span)?;
visited += 1;
let site = *sites
.get(&(
file,
owner,
field,
expression.span.start,
expression.span.end,
))
.ok_or_else(|| {
error(
expression.span,
"Parsed expression is absent from the source arena.",
)
})?;
if lowered.expressions[site].parent != parent {
return Err(error(
expression.span,
"Arena parent/child edge differs from original syntax.",
));
}
let mut child = |value: &'a super::Expression, edge| -> Result<(), Error> {
budget.lowering_entries(work.len() + 1, value.span)?;
work.push((file, owner, field, value, Some((site, edge))));
Ok(())
};
use super::ExpressionKind as E;
match &expression.kind {
E::List(items) => {
for (index, item) in items.iter().enumerate() {
child(item, Child::Item(index))?;
}
}
E::Unary { operand, .. } => child(operand, Child::Operand)?,
E::Binary { left, right, .. } => {
child(left, Child::Left)?;
child(right, Child::Right)?;
}
E::If {
condition,
when_true,
when_false,
} => {
child(condition, Child::Condition)?;
child(when_true, Child::WhenTrue)?;
child(when_false, Child::WhenFalse)?;
}
E::Call {
definition,
arguments,
} => {
let calculation = *globals
.get(definition.name.as_str())
.ok_or_else(|| error(definition.span, "Unknown syntax call target."))?;
for argument in arguments {
let parameter = *parameters
.get(&(calculation, argument.parameter.name.as_str()))
.ok_or_else(|| {
error(argument.parameter.span, "Unknown syntax call parameter.")
})?;
child(&argument.value, Child::Argument(parameter))?;
}
}
E::Sum { values, initial } => {
child(values, Child::Values)?;
child(initial, Child::Initial)?;
}
E::Power { base, .. } => child(base, Child::Base)?,
E::AsKind { value, .. } => child(value, Child::Value)?,
_ => {}
}
}
if visited != lowered.expressions.len() {
return Err(error(
Span::default(),
"Arena contains expressions absent from parsed syntax.",
));
}
Ok(())
}
fn insert(
&mut self,
key: OccurrenceKey,
location: Location,
witnesses: &BTreeMap<usize, Vec<lex::Token>>,
physical: &mut BTreeSet<(usize, usize, usize)>,
budget: &Budget,
) -> Result<(), Error> {
budget.lowering_occurrences(self.occurrences.len() + 1, location.span)?;
let text = checked_source(self.sources, location)?;
if self.names.get(&key.target).is_none_or(|name| name != text) {
return Err(error(
location.span,
"Token differs from its qualified target name.",
));
}
let tokens = witnesses
.get(&location.file)
.ok_or_else(|| error(location.span, "Missing lexical witness."))?;
let token = tokens
.binary_search_by_key(&location.span.start, |token| token.span.start)
.ok()
.map(|at| tokens[at]);
if token.is_none_or(|token| token.span != location.span || token.class != lex::Class::Word)
{
return Err(error(
location.span,
"Occurrence is not an original whole name token.",
));
}
if self.occurrences.contains_key(&key)
|| !physical.insert((location.file, location.span.start, location.span.end))
{
return Err(error(
location.span,
"Duplicate qualified occurrence or token ownership.",
));
}
self.occurrences.insert(key, location);
Ok(())
}
}
pub(super) fn checked_source(
sources: &BTreeMap<usize, String>,
location: Location,
) -> Result<&str, Error> {
if location.span.start >= location.span.end {
return Err(error(location.span, "Empty or reversed source span."));
}
sources
.get(&location.file)
.and_then(|source| source.get(location.span.start..location.span.end))
.ok_or_else(|| {
error(
location.span,
"Source span is outside its original UTF-8 file.",
)
})
}
fn contained(parent: Location, child: Location) -> Result<(), Error> {
if parent.file != child.file
|| child.span.start < parent.span.start
|| child.span.end > parent.span.end
{
return Err(error(
child.span,
"Source span crosses its containing syntax.",
));
}
Ok(())
}
#[cfg(test)]
mod tests;
@@ -0,0 +1,630 @@
use super::*;
use crate::language::ParseLimits;
const OA: &str = include_str!("../lower/fixtures/typed-oa.arcadia");
const ORDINARY: &str = include_str!("../lower/fixtures/ordinary-oa.arcadia");
fn budget() -> Budget {
Budget::new(super::super::budget::Limits::source8(
ParseLimits::UNLIMITED,
))
.unwrap()
}
fn ordinary() -> Project {
crate::language::parse(ORDINARY).unwrap()
}
fn sources() -> BTreeMap<usize, String> {
let split = OA.find("calculation combine").unwrap();
BTreeMap::from([
(7, OA[..split].replace('\n', "\r\n")),
(19, OA[split..].to_owned()),
])
}
fn raw(
sources: &BTreeMap<usize, String>,
budget: &mut Budget,
) -> (Vec<Unit>, BTreeMap<usize, Vec<lex::Token>>, Lowered) {
let mut units = Vec::new();
let mut witnesses = BTreeMap::new();
for (&file, source) in sources {
let tokens = lex::tokens(source, budget).unwrap();
let mut cursor = Cursor::new(source, &tokens, budget);
let mut declarations = Vec::new();
while cursor.cursor < tokens.len() {
declarations.push(declarations::parse(&mut cursor, 8, None).unwrap());
}
units.push(Unit { file, declarations });
witnesses.insert(file, tokens);
}
let lowered = lower::lower(&ordinary(), &units, budget).unwrap();
(units, witnesses, lowered)
}
#[test]
fn staged_index_preserves_complete_oa_sa_models_and_original_file_spans() {
for (layer, ordinary_source, typed) in [
(crate::Layer::Oa, ORDINARY, OA),
(
crate::Layer::Sa,
include_str!("../lower/fixtures/ordinary-sa.arcadia"),
include_str!("../lower/fixtures/typed-sa.arcadia"),
),
] {
let sources = BTreeMap::from([(37, typed.replace('\n', "\r\n"))]);
let skeleton = crate::language::parse(ordinary_source).unwrap();
let mut work = budget();
let (index, lowered) = Index::parse(&skeleton, &sources, &mut work).unwrap();
let mut actual = skeleton;
actual.definitions.extend(lowered.definitions);
actual.elements.extend(lowered.elements);
assert_eq!(actual, super::super::model_oracle::project(layer));
a::admit(&actual, a::AdmissionLimits::REFERENCE).unwrap();
assert_eq!(work.usage.bytes, sources[&37].len(), "one lexical pass");
for (key, at) in index.occurrences() {
assert_eq!(at.file, 37);
assert_eq!(
&sources[&37][at.span.start..at.span.end],
index.names[&key.target]
);
}
assert!(!index.expressions().is_empty());
}
}
#[test]
fn repeated_reordered_calls_have_stable_full_paths_and_four_labels() {
let expression = "@combine(b: @dispatch_time, a: @receive_time) // first 雪\n + @combine(a: @receive_time, b: @dispatch_time)";
let original = OA.replace("@combine(a: @receive_time, b: @dispatch_time)", expression);
let reordered = original.replace(
"@combine(b: @dispatch_time, a: @receive_time)",
"@combine(a: @receive_time, b: @dispatch_time)",
);
let first = BTreeMap::from([(7, original)]);
let second = BTreeMap::from([(7, reordered)]);
let (left, _) = Index::parse(&ordinary(), &first, &mut budget()).unwrap();
let (right, _) = Index::parse(&ordinary(), &second, &mut budget()).unwrap();
assert_eq!(
left.occurrences.keys().collect::<Vec<_>>(),
right.occurrences.keys().collect::<Vec<_>>()
);
assert_eq!(
left.expressions.keys().collect::<Vec<_>>(),
right.expressions.keys().collect::<Vec<_>>()
);
let labels: Vec<_> = left
.occurrences
.iter()
.filter(|(key, _)| key.role == TokenRole::Use(Role::ArgumentLabel))
.collect();
assert_eq!(labels.len(), 4);
let paths: BTreeSet<_> = labels
.iter()
.map(|(key, _)| key.path.as_ref().unwrap().edges())
.collect();
assert_eq!(
paths,
BTreeSet::from([vec![Child::Left], vec![Child::Right]])
);
let spans: BTreeSet<_> = labels
.iter()
.map(|(_, at)| (at.span.start, at.span.end))
.collect();
assert_eq!(spans.len(), 4);
}
#[test]
fn index_checks_every_expression_edge_and_keeps_override_fields_distinct() {
let source = OA
.replace("body $a + $b;", "body if(if(true, 1 < 2, false), sum([$a, abs(-$b)], 0 [@second]), as_kind(@duration, power(1 [@second], 2) / 1 [@second]));")
.replace("@receive_time = 1250 [@millisecond];", "@receive_time = @combine(a: 1 [@second], b: 2 [@second]); @dispatch_time = @combine(b: 2 [@second], a: 1 [@second]);");
let sources = BTreeMap::from([(17, source)]);
let (index, lowered) = Index::parse(&ordinary(), &sources, &mut budget()).unwrap();
let mut project = ordinary();
project.definitions.extend(lowered.definitions);
project.elements.extend(lowered.elements);
a::admit(&project, a::AdmissionLimits::REFERENCE).unwrap();
let edges: BTreeSet<_> = index
.expressions
.keys()
.flat_map(|key| key.path.edges())
.collect();
for edge in [
Child::Item(0),
Child::Item(1),
Child::Operand,
Child::Left,
Child::Right,
Child::Condition,
Child::WhenTrue,
Child::WhenFalse,
Child::Values,
Child::Initial,
Child::Base,
Child::Value,
] {
assert!(edges.contains(&edge), "missing {edge:?}");
}
assert!(edges.iter().any(|edge| matches!(edge, Child::Argument(_))));
let labels: Vec<_> = index
.occurrences
.keys()
.filter(|key| {
key.role == TokenRole::Use(Role::ArgumentLabel)
&& matches!(key.field, Some(Field::Override(_)))
})
.collect();
assert_eq!(labels.len(), 4);
assert_eq!(
labels
.iter()
.map(|key| key.field)
.collect::<BTreeSet<_>>()
.len(),
2
);
assert!(
labels
.iter()
.all(|key| key.path.as_ref().unwrap().edges().is_empty())
);
}
#[test]
fn arena_rejects_forward_cycles_and_cross_owner_field_file_parents() {
for damage in 0..6 {
let sources = sources();
let mut work = budget();
let (units, witnesses, mut lowered) = raw(&sources, &mut work);
let child = lowered
.expressions
.iter()
.position(|site| site.parent.is_some())
.unwrap();
let parent = lowered.expressions[child].parent.unwrap().0;
match damage {
0 => lowered.expressions[child].parent = Some((child, Child::Left)),
1 => lowered.expressions[parent].parent = Some((child, Child::Right)),
2 => lowered.expressions[child].owner = ordinary().id,
3 => lowered.expressions[child].field = Field::Predicate,
4 => lowered.expressions[child].file = 12345,
_ => lowered.expressions[child].parent = Some((parent, Child::Operand)),
}
assert!(
Index::build(
&ordinary(),
&sources,
&units,
&witnesses,
&lowered,
&mut work
)
.is_err(),
"damage {damage}"
);
}
}
#[test]
fn arena_rejects_duplicate_sites_occurrences_and_wrong_callee_labels() {
for damage in 0..4 {
let mut sources = sources();
// The other callee is fully valid and reuses both parameter identities
// and spellings. Catalog/token validation must not explain rejection.
sources.insert(
31,
r#"calculation other "Other" id "00000000-0000-0000-0000-00000000cafe" {
input a id "28ff40b0-1b8f-4735-8536-5a6ab21b56cb" : Quantity(@duration);
input b id "7a47cc1e-8ae5-4551-b2d8-8a68108e0976" : Quantity(@duration);
result Quantity(@duration); body $a + $b;
}"#
.into(),
);
let mut work = budget();
let (units, witnesses, mut lowered) = raw(&sources, &mut work);
let valid = Index::build(
&ordinary(),
&sources,
&units,
&witnesses,
&lowered,
&mut work,
)
.unwrap();
let mut expected_mismatch_span = None;
match damage {
0 => {
let site = &lowered.expressions[0];
lowered.expressions.push(lower::ExpressionSite {
file: site.file,
owner: site.owner,
field: site.field,
parent: site.parent,
span: site.span,
});
}
1 => {
let row = &lowered.occurrences[0];
lowered.occurrences.push(lower::Occurrence {
file: row.file,
owner: row.owner,
field: row.field,
expression: row.expression,
role: row.role,
target: row.target,
span: row.span,
});
}
2 => {
let label = lowered
.occurrences
.iter_mut()
.find(|row| row.role == Role::ArgumentLabel)
.unwrap();
let original = label.target;
let Target::Parameter {
calculation,
parameter,
} = &mut label.target
else {
panic!("argument label must target a parameter");
};
let other = Target::Parameter {
calculation: Id::from_u128(0xcafe),
parameter: *parameter,
};
assert_eq!(valid.names[&original], valid.names[&other]);
*calculation = Id::from_u128(0xcafe);
expected_mismatch_span = Some(label.span);
}
_ => {
let row = lowered
.occurrences
.iter_mut()
.find(|row| row.role == Role::OverrideTarget)
.unwrap();
row.field = Field::Override(ordinary().id);
}
}
let failure = Index::build(
&ordinary(),
&sources,
&units,
&witnesses,
&lowered,
&mut work,
)
.unwrap_err();
if let Some(span) = expected_mismatch_span {
assert_eq!(
failure.message,
"Argument label crosses its called calculation."
);
assert_eq!(failure.span, span);
}
}
}
#[test]
fn index_rejects_comment_partial_and_non_utf8_occurrence_spans() {
for damage in 0..3 {
let mut sources = sources();
let with_comment =
sources[&7].replace("subject @receive;", "subject @receive; // duration 雪\r\n");
sources.insert(7, with_comment);
let mut work = budget();
let (units, witnesses, mut lowered) = raw(&sources, &mut work);
let occurrence = lowered
.occurrences
.iter_mut()
.find(|row| row.file == 7 && &sources[&7][row.span.start..row.span.end] == "duration")
.unwrap();
match damage {
0 => {
let start = sources[&7].find("// duration").unwrap() + 3;
occurrence.span = Span {
start,
end: start + 8,
};
}
1 => occurrence.span.end -= 1,
_ => {
let start = sources[&7].find('雪').unwrap() + 1;
occurrence.span = Span {
start,
end: start + 1,
};
}
}
assert!(
Index::build(
&ordinary(),
&sources,
&units,
&witnesses,
&lowered,
&mut work
)
.is_err()
);
}
}
#[test]
fn index_has_shared_budget_and_never_stabilizes_missing_ids() {
let source = "calculation f \"F\" { result Number; body unbound; }";
let files = BTreeMap::from([(91, source.into())]);
let missing = Index::parse(&ordinary(), &files, &mut budget()).unwrap_err();
assert_eq!(missing.file, Some(91));
assert!(missing.error.message.contains("stabilization"));
let mut limits = super::super::budget::Limits::source8(ParseLimits::UNLIMITED);
limits.bytes = 1;
let mut work = Budget::new(limits).unwrap();
assert!(Index::parse(&ordinary(), &sources(), &mut work).is_err());
assert!(work.active().is_err());
}
fn on_small_stack(work: impl FnOnce() + Send + 'static) {
// Match the established core language_nesting_tests worker-stack request.
// This is a requested platform stack size, not a measured peak-byte claim.
std::thread::Builder::new()
.name("analysis-index-128k".into())
.stack_size(128 * 1024)
.spawn(work)
.unwrap()
.join()
.unwrap();
}
fn chain(first: Child, edges: usize) -> Path {
let mut path = Path::default();
for position in 0..edges {
path = Path(Some(Arc::new(Step {
parent: path,
edge: if position == 0 { first } else { Child::Operand },
})));
}
path
}
#[test]
fn maximum_index_paths_compare_lookup_debug_and_drop_on_128k_stack() {
on_small_stack(|| {
let frames = super::super::budget::Limits::source8(ParseLimits::UNLIMITED).frames;
assert_eq!(frames, 256);
let edges = frames - 1; // root has frame depth one
let mut work = budget();
work.frame(frames, Span::default()).unwrap();
let left = chain(Child::Left, edges);
let equal = chain(Child::Left, edges); // independent allocations
let right = chain(Child::Right, edges);
assert!(!Arc::ptr_eq(
left.0.as_ref().unwrap(),
equal.0.as_ref().unwrap()
));
assert_eq!(left, equal);
assert_eq!(left.partial_cmp(&equal), Some(Ordering::Equal));
// The difference is at the root-most edge: leaf-first walks every link.
assert_eq!(left.cmp(&right), Child::Left.cmp(&Child::Right));
assert_eq!(right.cmp(&left), left.cmp(&right).reverse());
assert!(Path::default() < left);
assert!(chain(Child::Operand, edges - 1) < chain(Child::Operand, edges));
let shared = left.clone();
assert!(Arc::ptr_eq(
left.0.as_ref().unwrap(),
shared.0.as_ref().unwrap()
));
let leaf = Arc::downgrade(left.0.as_ref().unwrap());
let key = |path| ExpressionKey {
owner: Id::from_u128(1),
field: Field::Body,
path,
};
let mut expressions = BTreeMap::new();
assert_eq!(expressions.insert(key(left.clone()), 1), None);
assert_eq!(expressions.insert(key(equal.clone()), 2), Some(1));
assert_eq!(expressions.insert(key(right.clone()), 3), None);
assert_eq!(expressions.get(&key(equal.clone())), Some(&2));
assert_eq!(expressions.get(&key(chain(Child::Right, edges))), Some(&3));
assert_eq!(expressions.len(), 2);
let occurrence = |path| OccurrenceKey {
owner: Id::from_u128(1),
field: Some(Field::Body),
path: Some(path),
role: TokenRole::Use(Role::ParameterUse),
target: Target::Parameter {
calculation: Id::from_u128(1),
parameter: a::ParameterId(Id::from_u128(2)),
},
};
let mut occurrences = BTreeSet::new();
assert!(occurrences.insert(occurrence(left.clone())));
assert!(!occurrences.insert(occurrence(equal.clone())));
assert!(occurrences.contains(&occurrence(chain(Child::Left, edges))));
let debug = format!("{left:?}");
assert_eq!(debug.matches("Operand").count(), edges - 1);
assert!(debug.ends_with("Left]"));
assert!(!format!("{expressions:?} {occurrences:?}").is_empty());
drop(expressions);
drop(occurrences);
drop(shared);
drop(left);
assert!(leaf.upgrade().is_none(), "maps/clones release their chain");
drop(equal);
drop(right);
// Dropping a child stops at a shared ancestor, which remains usable.
let parent = chain(Child::Left, edges - 1);
let ancestor = Arc::downgrade(parent.0.as_ref().unwrap());
let child = Path(Some(Arc::new(Step {
parent: parent.clone(),
edge: Child::Right,
})));
drop(child);
assert_eq!(parent.edges().len(), edges - 1);
drop(parent);
assert!(ancestor.upgrade().is_none());
// No other strong owners: final destruction must peel the full chain.
let unique = chain(Child::Left, edges);
let final_leaf = Arc::downgrade(unique.0.as_ref().unwrap());
drop(unique);
assert!(final_leaf.upgrade().is_none());
});
}
#[test]
fn deep_parse_index_plan_and_failed_build_cleanup_on_128k_stack() {
on_small_stack(|| {
use super::super::update::Plan;
let deep = format!("{}$a{}", "abs(".repeat(127), ")".repeat(127));
let source = format!(
r#"calculation f "Deep 雪" id "00000000-0000-0000-0000-000000000081" {{
input a id "00000000-0000-0000-0000-000000000083" : Number;
result Number; body {deep};
}}
calculation caller "Caller" id "00000000-0000-0000-0000-000000000082" {{
result Number; body @f(a: 1);
}}
"#
);
let sources = BTreeMap::from([(47, source.clone())]);
let skeleton = ordinary();
let mut work = budget();
let (index, lowered) = Index::parse(&skeleton, &sources, &mut work).unwrap();
assert_eq!(work.usage.bytes, source.len());
assert_eq!(index.expressions.len(), 130); // 128 unary/leaf + call/argument
assert_eq!(
index.expressions.keys().map(|k| k.path.edges().len()).max(),
Some(127)
);
let renames = BTreeMap::from([
(Target::Global(Id::from_u128(0x81)), "renamed_deep".into()),
(
Target::Parameter {
calculation: Id::from_u128(0x81),
parameter: a::ParameterId(Id::from_u128(0x83)),
},
"renamed_arg".into(),
),
]);
let usage = work.usage;
let plan = Plan::renames(&index, &renames, &work).unwrap();
assert_eq!(plan.edit_count(), 5); // declaration/call + input/use/label
let output = plan.apply(&sources, &work).unwrap();
assert_eq!(work.usage, usage);
// Independent complete original-token oracle; no index-derived offsets.
let expected_source = source
.replace("calculation f ", "calculation renamed_deep ")
.replace("input a id", "input renamed_arg id")
.replace("$a", "$renamed_arg")
.replace("@f(a: 1)", "@renamed_deep(renamed_arg: 1)");
assert_eq!(output, BTreeMap::from([(47, expected_source)]));
let (renamed_index, renamed) = Index::parse(&skeleton, &output, &mut budget()).unwrap();
assert_eq!(
index.occurrences.keys().collect::<Vec<_>>(),
renamed_index.occurrences.keys().collect::<Vec<_>>()
);
// Compare complete independently constructed models, including deepest
// parameter identity and the call's parameter-keyed argument map.
for (lowered, renamed) in [(lowered, false), (renamed, true)] {
let mut expected = skeleton.clone();
let parameter = a::ParameterId(Id::from_u128(0x83));
let mut expression = a::Expr::Parameter(parameter);
for _ in 0..127 {
expression = a::Expr::Unary {
op: a::UnaryOp::Absolute,
operand: Box::new(expression),
};
}
for (id, alias, name, parameters, expression) in [
(
0x81,
if renamed { "renamed_deep" } else { "f" },
"Deep 雪",
vec![a::Parameter {
id: parameter,
name: if renamed { "renamed_arg" } else { "a" }.into(),
value_type: a::AnalysisType::Scalar(a::ScalarType::Number),
}],
expression,
),
(
0x82,
"caller",
"Caller",
Vec::new(),
a::Expr::Call {
definition: a::DefinitionRef::Local(Id::from_u128(0x81)),
arguments: BTreeMap::from([(
parameter,
a::Expr::Number(crate::analysis_numeric::Exact::from_integer(1)),
)]),
},
),
] {
let id = Id::from_u128(id);
expected.definitions.insert(
id,
a::Definition {
id,
alias: alias.into(),
name: name.into(),
description: String::new(),
body: a::DefinitionBody::Calculation(a::CalculationDefinition {
parameters,
result: a::AnalysisType::Scalar(a::ScalarType::Number),
body: a::Binding::Expression(expression),
}),
},
);
}
let mut actual = skeleton.clone();
actual.definitions.extend(lowered.definitions);
actual.elements.extend(lowered.elements);
assert_eq!(actual, expected);
a::admit(&actual, a::AdmissionLimits::REFERENCE).unwrap();
drop(actual);
drop(expected);
}
drop(plan);
drop(renamed_index);
drop(index);
// Failure after the deepest keys have been allocated must clean up on
// the same stack; no modified Lowered escapes this private test seam.
let mut work = budget();
let (units, witnesses, mut lowered) = raw(&sources, &mut work);
let deepest = lowered
.expressions
.iter()
.find(|site| &sources[&site.file][site.span.start..site.span.end] == "$a")
.unwrap();
lowered.expressions.push(lower::ExpressionSite {
file: deepest.file,
owner: deepest.owner,
field: deepest.field,
parent: deepest.parent,
span: deepest.span,
});
let failure =
Index::build(&skeleton, &sources, &units, &witnesses, &lowered, &mut work).unwrap_err();
assert_eq!(failure.message, "Duplicate qualified expression path.");
lowered.expressions.pop();
let mut limits = super::super::budget::Limits::source8(ParseLimits::UNLIMITED);
limits.frames = 127;
let mut bounded = Budget::new(limits).unwrap();
let failure = Index::build(
&skeleton,
&sources,
&units,
&witnesses,
&lowered,
&mut bounded,
)
.unwrap_err();
assert!(failure.message.contains("syntax frames"));
assert!(bounded.active().is_err());
drop(lowered);
drop(witnesses);
drop(units);
let rejected = BTreeMap::from([(47, source.replace(&deep, &format!("abs({deep})")))]);
let failure = Index::parse(&skeleton, &rejected, &mut budget()).unwrap_err();
assert!(failure.error.message.contains("logical depth"));
});
}
@@ -34,7 +34,7 @@ pub(super) enum Category {
Calculation,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub(super) enum Target {
Global(Id),
Parameter {
@@ -43,7 +43,7 @@ pub(super) enum Target {
},
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub(super) enum Field {
Algebra,
Kind,
@@ -59,7 +59,7 @@ pub(super) enum Field {
Override(Id),
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub(super) enum Role {
Reference,
ParameterDeclaration,
@@ -70,7 +70,7 @@ pub(super) enum Role {
}
/// Stable structural edges, independent of alias spelling and argument order.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub(super) enum Child {
Item(usize),
Operand,
@@ -0,0 +1,400 @@
//! Private syntax-only complete-bundle witness. No model admission or edit authority.
//!
//! One scan charges original bytes and raw tokens (including trivia). Typed
//! syntax keeps its existing entry/reference/numeric charges. Ordinary syntax
//! charges one entry per project/declaration/include/description, field, and
//! value before allocation; array values already account for their list slots.
//! These private storage ceilings are independent of caller model record counts.
//! Ordinary references consume value storage, not ordered-model-step allowance.
//! Typed reference occurrences retain the caller max_step_references ceiling.
//! Only declaration graph/record counts are known here; semantic nested-model
//! records and ordinary ordered steps still require the later complete admission.
//! Metadata phase slots aggregate across all files: retained token-end indices,
//! value-end entries, projected value/field/header/declaration positions, lexical
//! owner slots and the file table. Checks do not recharge syntax or latch errors;
//! every failure must return immediately. Tokens themselves use the token bound.
//! This deliberately bounded staging policy is not calibrated public source8.
use super::{
budget::{Budget, Limits},
cursor::Cursor,
declarations, lex,
};
use crate::Id;
use crate::language::{
self, Ast, EditDeclaration, Error, Lexeme, ParseLimits, Parser, SourceUnit, Span, TokenClass,
};
use crate::source_project::{ProjectLimits, SourceBundle, SourceDiagnostic, traversal::Traversal};
use std::collections::BTreeMap;
struct ModelCounts {
limits: ProjectLimits,
graph: usize,
records: usize,
includes: usize,
}
pub(super) struct ParsedMixedSyntax<'s> {
bundle: &'s SourceBundle,
// Preserve caller policy for the later semantic pass. Syntax storage uses
// separate ceilings and cannot stand in for these graph/record/step limits.
limits: ProjectLimits,
files: Vec<FileSyntax<'s>>,
budget: Budget,
}
struct FileSyntax<'s> {
path: &'s str,
source: &'s str,
tokens: Vec<lex::Token>,
ordinary: SourceUnit,
typed: Vec<declarations::Declaration>,
positions: Vec<(Id, EditDeclaration)>,
// A lexical owner is an occurrence index, not only a possibly repeated UUID.
lexical_owners: Vec<Option<usize>>,
}
// Only Parser borrows this state; the stored witness has no self-reference.
pub(in crate::language) struct State<'a> {
source: &'a str,
tokens: &'a [lex::Token],
budget: &'a mut Budget,
model: &'a mut ModelCounts,
typed: Vec<declarations::Declaration>,
pub(in crate::language) value_ends: BTreeMap<usize, usize>,
pub(in crate::language) lexical_owners: Vec<Option<usize>>,
}
fn ordinary_error(error: super::Error) -> Error {
Error {
code: "invalid_source",
offset: error.span.start,
message: error.message,
}
}
fn lexeme(token: lex::Token) -> Lexeme {
Lexeme {
class: match token.class {
lex::Class::Word => TokenClass::Identifier,
lex::Class::Number => TokenClass::Number,
lex::Class::String => TokenClass::String,
lex::Class::Punctuation => TokenClass::Punctuation,
},
span: token.span,
}
}
impl State<'_> {
pub(in crate::language) fn token(&self, at: usize) -> Option<Lexeme> {
self.tokens.get(at).copied().map(lexeme)
}
pub(in crate::language) fn entry(&mut self, at: Span) -> Result<(), Error> {
self.budget.entry(at).map_err(ordinary_error)
}
pub(in crate::language) fn record(&mut self, graph: bool, at: Span) -> Result<(), Error> {
let records = self.model.records.checked_add(1);
let graph = self.model.graph.checked_add(usize::from(graph));
let (Some(records), Some(graph)) = (records, graph) else {
return Err(Error {
code: "model_limit",
offset: at.start,
message: "Declaration count overflow.".into(),
});
};
self.model
.limits
.model
.check_counts(graph, records, 0)
.map_err(|message| Error {
code: "model_limit",
offset: at.start,
message,
})?;
self.model.records = records;
self.model.graph = graph;
Ok(())
}
pub(in crate::language) fn include(&mut self, at: Span) -> Result<(), Error> {
if self.model.includes >= self.model.limits.max_include_edges {
return Err(Error {
code: "project_limit",
offset: at.start,
message: format!(
"Include graph exceeds {} edges.",
self.model.limits.max_include_edges
),
});
}
self.model.includes += 1;
Ok(())
}
pub(in crate::language) fn ordinary_number(
&self,
at: &mut usize,
) -> Result<serde_json::Value, Error> {
let first = self.tokens[*at];
let mut end = first.span.end;
let mut after = *at + 1;
// Match the old greedy numeric alphabet only across byte-adjacent raw
// tokens. This is consumption of the shared stream, not another scan.
while let Some(next) = self.tokens.get(after) {
if next.span.start != end
|| !self.source[next.span.start..next.span.end]
.bytes()
.all(|byte| byte.is_ascii_alphanumeric() || b"_.+-".contains(&byte))
{
break;
}
end = next.span.end;
after += 1;
}
let value: serde_json::Number = serde_json::from_str(&self.source[first.span.start..end])
.map_err(|_| Error {
code: "invalid_source",
offset: first.span.start,
message: "Expected a byte-adjacent finite JSON decimal number.".into(),
})?;
*at = after;
Ok(serde_json::Value::Number(value))
}
pub(in crate::language) fn typed_header(&self, at: usize, root: bool) -> bool {
let Some(token) = self.tokens.get(at) else {
return false;
};
if declarations::declaration_kind(&self.source[token.span.start..token.span.end]).is_none()
{
return false;
}
// A generic field named `unit` remains an ordinary field. A declaration
// header has an alias and display-name token; only that shape is nested.
root || (self
.tokens
.get(at + 1)
.is_some_and(|t| t.class == lex::Class::Word)
&& self
.tokens
.get(at + 2)
.is_some_and(|t| t.class == lex::Class::String))
}
pub(in crate::language) fn parse_typed(
&mut self,
at: &mut usize,
owner: Option<Id>,
) -> Result<(), Error> {
// Transfer the one active position into the typed consumer and back;
// neither consumer scans, clones a stream, nor runs concurrently.
self.record(true, self.tokens[*at].span)?;
let mut cursor = Cursor::new(self.source, self.tokens, self.budget);
cursor.cursor = *at;
let result = declarations::parse(&mut cursor, 8, owner);
*at = cursor.cursor;
let declaration = result.map_err(ordinary_error)?;
let missing = declaration.header.identity.value.is_none()
|| matches!(&declaration.body, declarations::Body::Calculation { parameters, .. }
if parameters.iter().any(|p| p.identity.value.is_none()));
if missing {
return Err(Error {
code: "invalid_source",
offset: declaration.span.start,
message: "Private mixed syntax requires explicit IDs.".into(),
});
}
self.typed.push(declaration);
Ok(())
}
}
pub(in crate::language) fn long_analysis_word(word: &str) -> bool {
matches!(
word,
"oa_engineering_property"
| "sa_engineering_property"
| "oa_analysis_check"
| "sa_analysis_check"
| "oa_analysis_case"
| "sa_analysis_case"
)
}
pub(super) fn parse(
bundle: &SourceBundle,
limits: ProjectLimits,
) -> Result<ParsedMixedSyntax<'_>, Vec<SourceDiagnostic>> {
let profile = Limits::source8(ParseLimits {
// Storage is not the caller's final model record/ordered-step accounting.
max_source_bytes: limits.model.max_source_bytes,
max_step_references: limits.model.max_step_references,
..ParseLimits::UNLIMITED
});
parse_with_profile(bundle, limits, profile)
}
fn located(path: &str, source: &str, error: Error) -> Vec<SourceDiagnostic> {
vec![SourceDiagnostic {
path: path.into(),
diagnostic: language::diagnostic(source, error),
related: Vec::new(),
}]
}
fn parse_with_profile(
bundle: &SourceBundle,
limits: ProjectLimits,
profile: Limits,
) -> Result<ParsedMixedSyntax<'_>, Vec<SourceDiagnostic>> {
let entry_source = bundle.files.get(&bundle.entry).map_or("", String::as_str);
let at_entry = |error| located(&bundle.entry, entry_source, ordinary_error(error));
let maximum = Limits::source8(ParseLimits {
max_source_bytes: limits.model.max_source_bytes,
max_step_references: limits.model.max_step_references,
..ParseLimits::UNLIMITED
});
if profile.bytes > maximum.bytes
|| profile.tokens > maximum.tokens
|| profile.entries > maximum.entries
|| profile.references > maximum.references
|| profile.frames > maximum.frames
|| profile.numeric_units > maximum.numeric_units
{
return Err(at_entry(super::error(
Span::default(),
"Mixed test profile may only tighten inherited limits.",
)));
}
let mut budget = Budget::new(profile).map_err(at_entry)?;
let bytes = bundle
.files
.values()
.try_fold(0usize, |n, s| n.checked_add(s.len()))
.ok_or_else(|| at_entry(super::error(Span::default(), "Source size overflow.")))?;
budget
.index_bytes(bytes, Span::default())
.map_err(at_entry)?;
let mut metadata = bundle.files.len();
budget
.lowering_entries(metadata, Span::default())
.map_err(at_entry)?;
// Tighter private byte/file-slot limits precede folded-path table and
// traversal allocations. Public Traversal's own validation order is intact.
let mut traversal = Traversal::new(bundle, limits)?;
let mut parsed = BTreeMap::new();
let mut model = ModelCounts {
limits,
graph: 0,
records: 0,
includes: 0,
};
while let Some(path) = traversal.next()? {
let (path, source) = bundle
.files
.get_key_value(&path)
.expect("traversal checked source");
let failure = |error| located(path, source, ordinary_error(error));
let tokens = lex::tokens(source, &mut budget).map_err(failure)?;
let mut parser = Parser {
source,
tokens: Vec::new(),
cursor: 0,
value_depth: 0,
limits: limits.model,
mixed: Some(State {
source,
tokens: &tokens,
budget: &mut budget,
model: &mut model,
typed: Vec::new(),
value_ends: BTreeMap::new(),
lexical_owners: Vec::new(),
}),
};
let ordinary = parser
.unit(path == &bundle.entry, true)
.map_err(|e| located(path, source, e))?;
let state = parser.mixed.take().expect("private parser state");
drop(parser);
// Projected ordinary AST nodes/fields are bounded before constructing a
// second position representation. Count references as slots, not syntax.
let projection_slots = ordinary_position_slots(&ordinary);
metadata = metadata
.checked_add(tokens.len())
.and_then(|n| n.checked_add(state.value_ends.len()))
.and_then(|n| n.checked_add(projection_slots))
.and_then(|n| n.checked_add(state.lexical_owners.len()))
.ok_or_else(|| failure(super::error(Span::default(), "Metadata size overflow.")))?;
state
.budget
.lowering_entries(metadata, Span::default())
.map_err(failure)?;
let positions = language::edit_positions_from_tokens(
source,
&ordinary,
|at| lexeme(tokens[at]),
tokens.len(),
Some(&state.value_ends),
)
.map_err(|e| {
vec![SourceDiagnostic {
path: path.clone(),
diagnostic: *e,
related: Vec::new(),
}]
})?;
for (raw, span) in &ordinary.includes {
traversal.include(path, raw, *span)?;
}
traversal.finish_file(path.clone());
let State {
typed,
lexical_owners,
..
} = state;
parsed.insert(
path.as_str(),
FileSyntax {
path,
source,
tokens,
ordinary,
typed,
positions,
lexical_owners,
},
);
}
traversal.finish()?;
budget.active().map_err(at_entry)?;
Ok(ParsedMixedSyntax {
bundle,
limits,
files: parsed.into_values().collect(),
budget,
})
}
fn ordinary_position_slots(unit: &SourceUnit) -> usize {
// Iterative accounting avoids adding a second recursive walk before the
// already depth-bounded position projection. Each reference is one value.
let mut slots = 0usize;
let mut stack = Vec::new();
for declaration in &unit.declarations {
slots = slots.saturating_add(7); // declaration/body + at most five header roles
slots = slots.saturating_add(declaration.fields.len());
stack.extend(declaration.fields.values());
}
while let Some(value) = stack.pop() {
slots = slots.saturating_add(1);
match value {
Ast::Array(items, _) => stack.extend(items),
Ast::Object(fields, _) => {
slots = slots.saturating_add(fields.len());
stack.extend(fields.values());
}
_ => {}
}
}
slots
}
#[cfg(test)]
mod tests;
File diff suppressed because it is too large Load Diff
@@ -349,6 +349,11 @@ impl Cursor<'_, '_> {
_ => 0,
};
if child_depth >= crate::analysis::MAX_EXPRESSION_DEPTH {
// The rejected parent still owns accepted, potentially depth-128
// children. Recursive derived cleanup here runs beneath the parser
// frames and can exhaust the bounded native stack before returning
// the depth error. Consume those children with an explicit worklist.
drop_rejected_kind(kind);
return Err(error(span, "Expression logical depth exceeds 128."));
}
Ok(Expression {
@@ -399,6 +404,42 @@ impl Cursor<'_, '_> {
self.node(kind, span)
}
}
fn drop_rejected_kind(kind: ExpressionKind) {
let mut pending = vec![kind];
while let Some(kind) = pending.pop() {
match kind {
ExpressionKind::Unary { operand, .. } => pending.push(operand.kind),
ExpressionKind::Binary { left, right, .. } => {
pending.push(left.kind);
pending.push(right.kind);
}
ExpressionKind::If {
condition,
when_true,
when_false,
} => {
pending.push(condition.kind);
pending.push(when_true.kind);
pending.push(when_false.kind);
}
ExpressionKind::List(values) => pending.extend(values.into_iter().map(|v| v.kind)),
ExpressionKind::Call { arguments, .. } => {
pending.extend(arguments.into_iter().map(|a| a.value.kind));
}
ExpressionKind::Sum { values, initial } => {
pending.push(values.kind);
pending.push(initial.kind);
}
ExpressionKind::Power { base, .. } => pending.push(base.kind),
ExpressionKind::AsKind { value, .. } => pending.push(value.kind),
ExpressionKind::Boolean(_)
| ExpressionKind::Number(_)
| ExpressionKind::Quantity { .. }
| ExpressionKind::Property(_)
| ExpressionKind::Parameter(_) => {}
}
}
}
fn comparison(op: BinaryOp) -> bool {
matches!(
op,
@@ -108,6 +108,49 @@ fn flat_binary_depth_and_malformed_constructor_retain_bounded_completed_children
assert_eq!(number(&expression("1")), "1");
}
#[test]
fn every_parent_shape_rejects_depth129_and_cleans_children_on_128k_stack() {
std::thread::Builder::new()
.name("analysis-syntax-rejection-128k".into())
.stack_size(128 * 1024)
.spawn(|| {
// Rotate through every recursive syntax shape, so the rejected
// parent's complete child tree also exercises every cleanup arm.
let wrap = |shape, child: &str| match shape {
0 => format!("abs({child})"),
1 => format!("({child}) + 1"),
2 => format!("if(true, {child}, 0)"),
3 => format!("[{child}]"),
4 => format!("@f(a: {child})"),
5 => format!("sum({child}, 0)"),
6 => format!("power({child}, 1)"),
7 => format!("as_kind(@k, {child})"),
_ => unreachable!(),
};
let mut deep = "1".to_owned();
for layer in 0..127 {
deep = wrap(layer % 8, &deep);
}
assert_eq!(expression(&deep).depth, 128);
for shape in 0..8 {
let source = wrap(shape, &deep);
let failure = parse(&source, &mut budget()).unwrap_err();
assert_eq!(failure.message, "Expression logical depth exceeds 128.");
assert_eq!(
failure.span,
Span {
start: 0,
end: source.len()
}
);
assert_eq!(number(&expression("1")), "1");
}
})
.unwrap()
.join()
.unwrap();
}
#[test]
fn significant_ratio_and_quantity_trivia_preserve_original_spans() {
let source = "ratio // explanation café\n ( - // sign\n 3, // denominator\n 4 ) // measurement\n [@millisecond]";
@@ -0,0 +1,316 @@
//! Simultaneous typed token edits. This is a staged plan, not source acceptance:
//! the bundle owner must merge ordinary edits, reparse/recompile the exact bytes,
//! admit once and compare the complete intended Project before installation.
use super::index::{Index, Location, OccurrenceKey, checked_source};
use super::lower::Target;
use super::{Budget, Error, Span, error};
use std::collections::{BTreeMap, BTreeSet};
#[derive(Debug, Clone, PartialEq, Eq)]
struct Edit {
key: OccurrenceKey,
location: Location,
}
#[derive(Debug)]
pub(super) struct Plan<'i, 's> {
index: &'i Index<'s>,
names: BTreeMap<Target, String>,
edits: Vec<Edit>,
}
impl<'i, 's> Plan<'i, 's> {
/// All names are the desired final names. Validate the final namespaces,
/// rather than applying one rename at a time: alias/parameter swaps are legal.
pub fn renames(
index: &'i Index<'s>,
renames: &BTreeMap<Target, String>,
budget: &Budget,
) -> Result<Self, Error> {
let plan = Self::prepare(index, renames, budget)?;
plan.validate(index.sources(), budget)?;
plan.output_lengths(index.sources(), budget)?;
Ok(plan)
}
fn validate_names(
index: &Index<'_>,
renames: &BTreeMap<Target, String>,
budget: &Budget,
) -> Result<(), Error> {
budget.active()?;
budget.lowering_entries(renames.len(), Span::default())?;
let mut bytes = 0usize;
for (target, name) in renames {
if !index.names().contains_key(target) {
return Err(error(
Span::default(),
"Rename has no qualified source target.",
));
}
if !crate::valid_alias(name) {
return Err(error(
Span::default(),
"Invalid alias or scoped parameter name.",
));
}
bytes = bytes
.checked_add(name.len())
.ok_or_else(|| error(Span::default(), "Rename size overflow."))?;
budget.index_bytes(bytes, Span::default())?;
}
let mut final_names = BTreeSet::new();
for (target, original) in index.names() {
let name = renames.get(target).unwrap_or(original);
let scope = match target {
Target::Global(_) => None,
Target::Parameter { calculation, .. } => Some(*calculation),
};
budget.lowering_entries(final_names.len() + 1, Span::default())?;
if !final_names.insert((scope, name.as_str())) {
return Err(error(
Span::default(),
"Rename creates a duplicate name in its namespace.",
));
}
}
Ok(())
}
fn prepare(
index: &'i Index<'s>,
renames: &BTreeMap<Target, String>,
budget: &Budget,
) -> Result<Self, Error> {
// Validate record/name storage before copying any owned final names.
Self::validate_names(index, renames, budget)?;
let mut edits = Vec::new();
for (key, location) in index.occurrences() {
if renames
.get(&key.target)
.is_some_and(|name| index.names().get(&key.target) != Some(name))
{
budget.lowering_occurrences(edits.len() + 1, location.span)?;
edits.push(Edit {
key: key.clone(),
location: *location,
});
}
}
edits.sort_by_key(|edit| {
(
edit.location.file,
edit.location.span.start,
edit.location.span.end,
)
});
Ok(Self {
index,
names: renames.clone(),
edits,
})
}
pub fn edit_count(&self) -> usize {
self.edits.len()
}
/// Recheck the complete original witness and complete rename intent.
/// A changed unrelated file is stale too. This does not construct output.
fn validate(&self, current: &BTreeMap<usize, String>, budget: &Budget) -> Result<(), Error> {
budget.active()?;
if current != self.index.sources() {
return Err(error(
Span::default(),
"Source index is stale: original file set or bytes changed.",
));
}
budget.lowering_entries(current.len(), Span::default())?;
budget.lowering_occurrences(self.edits.len(), Span::default())?;
let mut previous: Option<Location> = None;
let mut seen = BTreeSet::new();
for edit in &self.edits {
let at = edit.location;
if self.index.occurrences().get(&edit.key) != Some(&at) || !seen.insert(&edit.key) {
return Err(error(at.span, "Forged or duplicate edit occurrence."));
}
let old = checked_source(current, at)?;
if self
.index
.names()
.get(&edit.key.target)
.is_none_or(|name| name != old)
{
return Err(error(at.span, "Stale original token spelling."));
}
self.names
.get(&edit.key.target)
.ok_or_else(|| error(at.span, "Edit has no final target name."))?;
if previous.is_some_and(|prior| {
prior.file > at.file || (prior.file == at.file && prior.span.end > at.span.start)
}) {
return Err(error(at.span, "Overlapping or unordered source edits."));
}
previous = Some(at);
}
for (key, at) in self.index.occurrences() {
let required = self
.names
.get(&key.target)
.is_some_and(|name| self.index.names().get(&key.target) != Some(name));
if required != seen.contains(key) {
return Err(error(
at.span,
"Edit set differs from the complete qualified rename plan.",
));
}
}
Ok(())
}
/// Standalone output preflight, after ownership validation. Composition
/// defers this hypothetical typed-only size to its future combined emitter.
fn output_lengths(
&self,
current: &BTreeMap<usize, String>,
budget: &Budget,
) -> Result<BTreeMap<usize, usize>, Error> {
let mut lengths: BTreeMap<_, _> = current
.iter()
.map(|(&file, source)| (file, source.len()))
.collect();
for edit in &self.edits {
let at = edit.location;
let length = lengths.get_mut(&at.file).expect("validated edit file");
*length = length
.checked_sub(at.span.end - at.span.start)
.and_then(|size| size.checked_add(self.names[&edit.key.target].len()))
.ok_or_else(|| error(at.span, "Updated source size overflow."))?;
}
let mut total = 0usize;
for length in lengths.values() {
total = total
.checked_add(*length)
.ok_or_else(|| error(Span::default(), "Updated bundle size overflow."))?;
budget.index_bytes(total, Span::default())?;
}
Ok(lengths)
}
pub fn apply(
&self,
current: &BTreeMap<usize, String>,
budget: &Budget,
) -> Result<BTreeMap<usize, String>, Error> {
self.validate(current, budget)?;
let lengths = self.output_lengths(current, budget)?;
let mut result = BTreeMap::new();
let mut edits = self.edits.iter().peekable();
for (&file, source) in current {
let mut output = String::with_capacity(lengths[&file]);
let mut start = 0;
while edits.peek().is_some_and(|edit| edit.location.file == file) {
let edit = edits.next().expect("peeked edit");
output.push_str(&source[start..edit.location.span.start]);
output.push_str(&self.names[&edit.key.target]);
start = edit.location.span.end;
}
output.push_str(&source[start..]);
result.insert(file, output);
}
Ok(result)
}
}
/// Private typed-only composition planning. The caller must supply its inherited
/// parse Budget; this type keeps that same immutable budget through export.
/// It does not establish the future complete bundle/file-table/parse-session
/// witness. In particular, an Index currently does not store its parse profile.
pub(super) struct Composition<'b, 'i, 's> {
plan: Plan<'i, 's>,
budget: &'b Budget,
}
impl std::fmt::Debug for Composition<'_, '_, '_> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Composition")
.field("plan", &self.plan)
.finish_non_exhaustive()
}
}
impl<'b, 'i, 's> Composition<'b, 'i, 's> {
pub fn renames(
index: &'i Index<'s>,
renames: &BTreeMap<Target, String>,
inherited: &'b Budget,
) -> Result<Self, Error> {
let plan = Plan::prepare(index, renames, inherited)?;
plan.validate(index.sources(), inherited)?;
Ok(Self {
plan,
budget: inherited,
})
}
/// Export the entire validated plan against exact original numeric file keys
/// and bytes. There is no output allocation, application or subset-accept API.
/// A future collector must check this actual index witness and the complete
/// combined output bounds before applying either producer's edits once.
pub fn original_edits(
&self,
current: &BTreeMap<usize, String>,
) -> Result<OriginalEdits<'_, 'i, 's>, Error> {
Plan::validate_names(self.plan.index, &self.plan.names, self.budget)?;
self.plan.validate(current, self.budget)?;
Ok(OriginalEdits { plan: &self.plan })
}
}
#[derive(Debug)]
pub(super) struct OriginalEdits<'p, 'i, 's> {
plan: &'p Plan<'i, 's>,
}
impl<'s> OriginalEdits<'_, '_, 's> {
pub fn index(&self) -> &Index<'s> {
self.plan.index
}
pub fn len(&self) -> usize {
self.plan.edits.len()
}
pub fn is_empty(&self) -> bool {
self.plan.edits.is_empty()
}
pub fn iter(&self) -> impl ExactSizeIterator<Item = OriginalEdit<'_>> {
self.plan.edits.iter().map(|edit| OriginalEdit {
key: &edit.key,
location: edit.location,
original: &self.plan.index.sources()[&edit.location.file]
[edit.location.span.start..edit.location.span.end],
replacement: &self.plan.names[&edit.key.target],
})
}
}
/// A borrowed inspection record, never independent edit authority. Only an
/// intact OriginalEdits batch can carry this producer's complete rename intent.
#[derive(Debug)]
pub(super) struct OriginalEdit<'p> {
key: &'p OccurrenceKey,
location: Location,
original: &'p str,
replacement: &'p str,
}
impl OriginalEdit<'_> {
pub fn key(&self) -> &OccurrenceKey {
self.key
}
pub fn location(&self) -> Location {
self.location
}
pub fn original(&self) -> &str {
self.original
}
pub fn replacement(&self) -> &str {
self.replacement
}
}
#[cfg(test)]
mod tests;
@@ -0,0 +1,744 @@
use super::super::budget::Limits;
use super::*;
use crate::analysis as a;
use crate::analysis_numeric::Exact;
use crate::language::ParseLimits;
use crate::{Id, Layer, Project};
const DEFINITIONS: &str = "// 雪 @f(a:) $a: comment spellings stay exact\r\n\
calculation f \"f a b\" id \"00000000-0000-0000-0000-000000000100\" {\r\n\
input a id \"00000000-0000-0000-0000-000000000200\" : Number;\r\n\
input b id \"00000000-0000-0000-0000-000000000201\" : Number;\r\n\
result Number; body $a + $b;\r\n}\r\n\
calculation g \"g a b\" id \"00000000-0000-0000-0000-000000000200\" {\r\n\
input a id \"00000000-0000-0000-0000-000000000200\" : Number;\r\n\
input b id \"00000000-0000-0000-0000-000000000201\" : Number;\r\n\
result Number; body $a + $b;\r\n}\r\n";
const CALLER: &str = "calculation caller \"Caller\" id \"00000000-0000-0000-0000-000000000300\" {\n\
result Number; body @f(b: 2, a: 1) // first 雪 @f(a:)\n\
+ @f(a: 3, b: 4) // second\n\
+ @g(a: 5, b: 6);\n}\n";
fn budget() -> Budget {
Budget::new(Limits::source8(ParseLimits::UNLIMITED)).unwrap()
}
fn global(id: u128) -> Target {
Target::Global(Id::from_u128(id))
}
fn parameter(calculation: u128, parameter: u128) -> Target {
Target::Parameter {
calculation: Id::from_u128(calculation),
parameter: a::ParameterId(Id::from_u128(parameter)),
}
}
fn sources() -> BTreeMap<usize, String> {
BTreeMap::from([(11, DEFINITIONS.into()), (23, CALLER.into())])
}
fn skeleton() -> Project {
let mut project = super::super::model_oracle::project(Layer::Oa);
project.definitions.clear();
project
.elements
.retain(|_, element| !element.kind().is_analysis());
project
}
fn binary(left: a::Expr, right: a::Expr) -> a::Expr {
a::Expr::Binary {
op: a::BinaryOp::Add,
left: Box::new(left),
right: Box::new(right),
}
}
fn expected(renamed: bool) -> Project {
// Independent model construction: no parsed AST, source index, writer or
// generated edit participates in this expected complete Project.
let mut project = skeleton();
let p = |id| a::ParameterId(Id::from_u128(id));
for (id, alias, name) in [
(0x100, if renamed { "g" } else { "f" }, "f a b"),
(0x200, if renamed { "f" } else { "g" }, "g a b"),
] {
let names = if renamed && id == 0x100 {
["b", "a"]
} else {
["a", "b"]
};
let body = a::CalculationDefinition {
parameters: [(0x200, names[0]), (0x201, names[1])]
.into_iter()
.map(|(id, name)| a::Parameter {
id: p(id),
name: name.into(),
value_type: a::AnalysisType::Scalar(a::ScalarType::Number),
})
.collect(),
result: a::AnalysisType::Scalar(a::ScalarType::Number),
body: a::Binding::Expression(binary(
a::Expr::Parameter(p(0x200)),
a::Expr::Parameter(p(0x201)),
)),
};
let id = Id::from_u128(id);
project.definitions.insert(
id,
a::Definition {
id,
alias: alias.into(),
name: name.into(),
description: String::new(),
body: a::DefinitionBody::Calculation(body),
},
);
}
let call = |id, x, y| a::Expr::Call {
definition: a::DefinitionRef::Local(Id::from_u128(id)),
arguments: BTreeMap::from([
(p(0x200), a::Expr::Number(Exact::from_integer(x))),
(p(0x201), a::Expr::Number(Exact::from_integer(y))),
]),
};
let id = Id::from_u128(0x300);
project.definitions.insert(
id,
a::Definition {
id,
alias: "caller".into(),
name: "Caller".into(),
description: String::new(),
body: a::DefinitionBody::Calculation(a::CalculationDefinition {
parameters: Vec::new(),
result: a::AnalysisType::Scalar(a::ScalarType::Number),
body: a::Binding::Expression(binary(
binary(call(0x100, 1, 2), call(0x100, 3, 4)),
call(0x200, 5, 6),
)),
}),
},
);
project
}
fn merge(mut project: Project, lowered: super::super::lower::Lowered) -> Project {
project.definitions.extend(lowered.definitions);
project.elements.extend(lowered.elements);
project
}
fn rename_map() -> BTreeMap<Target, String> {
BTreeMap::from([
(global(0x100), "g".into()),
(global(0x200), "f".into()),
(parameter(0x100, 0x200), "b".into()),
(parameter(0x100, 0x201), "a".into()),
])
}
#[test]
fn simultaneous_global_and_scoped_swaps_match_original_token_and_whole_model_oracles() {
let sources = sources();
let mut work = budget();
let (index, lowered) = Index::parse(&skeleton(), &sources, &mut work).unwrap();
assert_eq!(merge(skeleton(), lowered), expected(false));
let plan = Plan::renames(&index, &rename_map(), &work).unwrap();
// List expected edits against the original authored bytes independently of
// the index. Full contexts deliberately exclude equal strings/comments.
let mut edits = Vec::new();
let mut add = |file, context: &str, token: &str, replacement: &str| {
let source = &sources[&file];
let context_start = source.find(context).unwrap();
let start = context_start + context.find(token).unwrap();
edits.push((file, start, start + token.len(), replacement.to_owned()));
};
add(11, "calculation f ", "f", "g");
add(11, "calculation g ", "g", "f");
add(11, "input a id", "a", "b");
add(11, "input b id", "b", "a");
add(11, "body $a + $b;", "a", "b");
add(11, "body $a + $b;", "b;", "a;");
// The last oracle edit above includes ';' to disambiguate 'b' in 'body';
// restrict it back to the exact name token for the span oracle.
edits.last_mut().unwrap().2 -= 1;
edits.last_mut().unwrap().3.pop();
for call in ["@f(b: 2, a: 1)", "@f(a: 3, b: 4)"] {
add_call_edits(&sources, &mut edits, call);
}
let start = CALLER.find("@g(a: 5, b: 6)").unwrap() + 1;
edits.push((23, start, start + 1, "f".into()));
edits.sort_by_key(|edit| (edit.0, edit.1));
assert_eq!(plan.edit_count(), 13);
assert_eq!(
plan.edits
.iter()
.map(|edit| (
edit.location.file,
edit.location.span.start,
edit.location.span.end,
plan.names[&edit.key.target].clone()
))
.collect::<Vec<_>>(),
edits
);
let mut exact = BTreeMap::new();
for (&file, source) in &sources {
let mut output = source.clone();
for (_, start, end, value) in edits.iter().filter(|edit| edit.0 == file).rev() {
output.replace_range(*start..*end, value);
}
exact.insert(file, output);
}
let output = plan.apply(&sources, &work).unwrap();
assert_eq!(output, exact);
assert!(output[&11].contains("// 雪 @f(a:) $a: comment spellings stay exact\r\n"));
assert!(output[&23].contains("// first 雪 @f(a:)\n"));
assert!(output[&11].contains("\"f a b\""));
let (_, lowered) = Index::parse(&skeleton(), &output, &mut budget()).unwrap();
let actual = merge(skeleton(), lowered);
assert_eq!(actual, expected(true));
a::admit(&actual, a::AdmissionLimits::REFERENCE).unwrap();
}
fn add_call_edits(
sources: &BTreeMap<usize, String>,
edits: &mut Vec<(usize, usize, usize, String)>,
call: &str,
) {
let start = sources[&23].find(call).unwrap();
for (offset, replacement) in [
(1, "g"),
(call.find("a:").unwrap(), "b"),
(call.find("b:").unwrap(), "a"),
] {
edits.push((23, start + offset, start + offset + 1, replacement.into()));
}
}
#[test]
fn alias_edits_preserve_unit_type_override_tokens_and_unrelated_authored_bytes() {
let source = include_str!("../lower/fixtures/typed-oa.arcadia")
.replace("750 [@millisecond]", "750 // exact 雪\r\n [@millisecond]");
let files = BTreeMap::from([(59, source.clone())]);
let mut work = budget();
let (index, _) = Index::parse(&skeleton(), &files, &mut work).unwrap();
let mut expected_project = super::super::model_oracle::project(Layer::Oa);
let id = |name: &str| {
index
.names()
.iter()
.find(|(target, alias)| matches!(target, Target::Global(_)) && alias.as_str() == name)
.map(|(target, _)| *target)
.unwrap()
};
let renames = BTreeMap::from([
(id("duration"), "time_kind".into()),
(id("millisecond"), "ms_unit".into()),
(id("receive_time"), "arrival".into()),
]);
let plan = Plan::renames(&index, &renames, &work).unwrap();
let output = plan.apply(&files, &work).unwrap();
let expected_source = source
.replace("quantity_kind duration ", "quantity_kind time_kind ")
.replace("@duration", "@time_kind")
.replace("unit millisecond ", "unit ms_unit ")
.replace("@millisecond", "@ms_unit")
.replace("oa_property receive_time ", "oa_property arrival ")
.replace("@receive_time", "@arrival");
assert_eq!(output[&59], expected_source);
for definition in expected_project.definitions.values_mut() {
if let Some(name) = renames.get(&Target::Global(definition.id)) {
definition.alias = name.clone();
}
}
for element in expected_project.elements.values_mut() {
if let Some(name) = renames.get(&Target::Global(element.id)) {
element.alias = name.clone();
}
}
let (_, lowered) = Index::parse(&skeleton(), &output, &mut budget()).unwrap();
assert_eq!(merge(skeleton(), lowered), expected_project);
}
#[test]
fn rename_rejects_unknown_invalid_and_colliding_final_names_but_allows_noop() {
let sources = sources();
let work = &mut budget();
let (index, _) = Index::parse(&skeleton(), &sources, work).unwrap();
for renames in [
BTreeMap::from([(global(999), "new".into())]),
BTreeMap::from([(global(0x100), "bad name".into())]),
BTreeMap::from([(global(0x100), "g".into())]),
BTreeMap::from([(parameter(0x100, 0x200), "b".into())]),
] {
assert!(Plan::renames(&index, &renames, work).is_err());
}
let plan = Plan::renames(&index, &BTreeMap::from([(global(0x100), "f".into())]), work).unwrap();
assert_eq!(plan.edit_count(), 0);
assert_eq!(plan.apply(&sources, work).unwrap(), sources);
}
#[test]
fn stale_file_sets_bytes_forged_spans_and_duplicate_edits_fail_before_output() {
let sources = sources();
let mut work = budget();
let (index, _) = Index::parse(&skeleton(), &sources, &mut work).unwrap();
for damage in 0..6 {
let mut current = sources.clone();
let mut plan = Plan::renames(&index, &rename_map(), &work).unwrap();
match damage {
0 => current
.get_mut(&23)
.unwrap()
.push_str("// unrelated change"),
1 => {
current.remove(&23);
}
2 => {
current.insert(99, String::new());
}
3 => plan.edits[0].location.span.start += 1,
4 => plan.edits.insert(1, plan.edits[0].clone()),
_ => {
plan.edits.pop();
}
}
assert!(plan.apply(&current, &work).is_err(), "damage {damage}");
}
}
#[test]
fn output_and_plan_limits_are_aggregate_and_do_not_reset_parse_work() {
let sources = sources();
let mut work = budget();
let (index, _) = Index::parse(&skeleton(), &sources, &mut work).unwrap();
let usage = work.usage;
let numeric = work.numeric_units();
let plan = Plan::renames(&index, &rename_map(), &work).unwrap();
let output = plan.apply(&sources, &work).unwrap();
assert_eq!(work.usage, usage);
assert_eq!(work.numeric_units(), numeric);
let bytes = output.values().map(String::len).sum();
let mut limits = Limits::source8(ParseLimits::UNLIMITED);
limits.bytes = bytes;
assert!(plan.apply(&sources, &Budget::new(limits).unwrap()).is_ok());
limits.bytes -= 1;
assert!(plan.apply(&sources, &Budget::new(limits).unwrap()).is_err());
limits = Limits::source8(ParseLimits::UNLIMITED);
limits.references = 0;
limits.entries = 0;
assert!(Plan::renames(&index, &rename_map(), &Budget::new(limits).unwrap()).is_err());
}
#[test]
fn composition_exports_complete_borrowed_swap_witnesses_and_empty_noops() {
use super::super::index::TokenRole;
use super::super::lower::{Child, Field, Role};
type Key = (Id, Option<Field>, Option<Vec<Child>>, TokenRole, Target);
type Row = (Location, String, String, Key);
let sources = sources();
let mut work = budget();
let (index, _) = Index::parse(&skeleton(), &sources, &mut work).unwrap();
let usage = work.usage;
let numeric = work.numeric_units();
let composition = Composition::renames(&index, &rename_map(), &work).unwrap();
assert!(std::ptr::eq(composition.budget, &work));
// A byte-identical snapshot is valid; records still borrow the actual index.
let batch = composition.original_edits(&sources.clone()).unwrap();
assert!(std::ptr::eq(batch.index(), &index));
assert_eq!(batch.len(), 13);
assert!(!batch.is_empty());
assert_eq!(batch.iter().len(), 13);
// Independently authored occurrence identities and full source contexts.
// No index offset/path or generated output defines this oracle.
let mut expected: Vec<Row> = Vec::new();
let mut add = |file, context: &str, prefix: &str, old: &str, new: &str, key: Key| {
let start = sources[&file].find(context).unwrap() + prefix.len();
assert_eq!(&sources[&file][start..start + old.len()], old);
expected.push((
Location {
file,
span: Span {
start,
end: start + old.len(),
},
},
old.into(),
new.into(),
key,
));
};
for (id, old, new, context) in [
(0x100, "f", "g", "calculation f "),
(0x200, "g", "f", "calculation g "),
] {
add(
11,
context,
"calculation ",
old,
new,
(
Id::from_u128(id),
None,
None,
TokenRole::Declaration,
global(id),
),
);
}
for (id, old, new, context, edge, prefix) in [
(0x200, "a", "b", "input a id", Child::Left, "body $"),
(0x201, "b", "a", "input b id", Child::Right, "body $a + $"),
] {
let target = parameter(0x100, id);
add(
11,
context,
"input ",
old,
new,
(
Id::from_u128(0x100),
Some(Field::Input(a::ParameterId(Id::from_u128(id)))),
None,
TokenRole::Use(Role::ParameterDeclaration),
target,
),
);
add(
11,
"body $a + $b;",
prefix,
old,
new,
(
Id::from_u128(0x100),
Some(Field::Body),
Some(vec![edge]),
TokenRole::Use(Role::ParameterUse),
target,
),
);
}
for (context, path) in [
("@f(b: 2, a: 1)", vec![Child::Left, Child::Left]),
("@f(a: 3, b: 4)", vec![Child::Left, Child::Right]),
] {
add(
23,
context,
"@",
"f",
"g",
(
Id::from_u128(0x300),
Some(Field::Body),
Some(path.clone()),
TokenRole::Use(Role::CallTarget),
global(0x100),
),
);
for (id, old, new) in [(0x200, "a", "b"), (0x201, "b", "a")] {
let offset = context.find(&format!("{old}:")).unwrap();
add(
23,
context,
&context[..offset],
old,
new,
(
Id::from_u128(0x300),
Some(Field::Body),
Some(path.clone()),
TokenRole::Use(Role::ArgumentLabel),
parameter(0x100, id),
),
);
}
}
add(
23,
"@g(a: 5, b: 6)",
"@",
"g",
"f",
(
Id::from_u128(0x300),
Some(Field::Body),
Some(vec![Child::Right]),
TokenRole::Use(Role::CallTarget),
global(0x200),
),
);
expected.sort_by_key(|row| (row.0.file, row.0.span.start, row.0.span.end));
let actual: Vec<Row> = batch
.iter()
.map(|edit| {
let key = edit.key();
(
edit.location(),
edit.original().into(),
edit.replacement().into(),
(
key.owner,
key.field,
key.path.as_ref().map(|path| path.edges()),
key.role,
key.target,
),
)
})
.collect();
assert_eq!(actual, expected);
let noop = Composition::renames(
&index,
&BTreeMap::from([
(global(0x100), "f".into()),
(parameter(0x200, 0x200), "a".into()),
]),
&work,
)
.unwrap();
let empty = noop.original_edits(&sources).unwrap();
assert!(empty.is_empty());
assert_eq!(empty.len(), 0);
assert_eq!(empty.iter().count(), 0);
assert_eq!(work.usage, usage);
assert_eq!(work.numeric_units(), numeric);
}
#[test]
fn composition_rejects_stale_or_damaged_complete_batches_before_export() {
let mut sources = sources();
sources.insert(99, "// unrelated 雪\r\n".into());
let original = sources.clone();
let mut work = budget();
let (index, _) = Index::parse(&skeleton(), &sources, &mut work).unwrap();
for damage in 0..10 {
let mut current = sources.clone();
let mut composition = Composition::renames(&index, &rename_map(), &work).unwrap();
match damage {
0 => current.get_mut(&99).unwrap().push_str("changed"),
1 => {
current.remove(&23);
}
2 => {
current.insert(101, String::new());
}
3 => composition.plan.edits[0].location.span.start += 1,
4 => composition
.plan
.edits
.insert(1, composition.plan.edits[0].clone()),
5 => {
composition.plan.edits.pop();
}
6 => composition.plan.edits.swap(0, 1),
7 => {
composition
.plan
.names
.insert(global(0x100), "bad name".into());
}
8 => {
composition.plan.names.insert(global(0x100), "f".into());
}
_ => {
composition.plan.names.insert(global(999), "missing".into());
}
}
let before = current.clone();
assert!(
composition.original_edits(&current).is_err(),
"damage {damage}"
);
assert_eq!(current, before);
}
assert_eq!(sources, original);
let failure = work.frame(257, Span::default()).unwrap_err();
assert_eq!(
Composition::renames(&index, &rename_map(), &work).unwrap_err(),
failure
);
}
#[test]
fn composition_defers_only_typed_output_size_with_independent_shrink_arithmetic() {
let sources = sources();
let cap = sources.values().map(String::len).sum();
let mut limits = Limits::source8(ParseLimits::UNLIMITED);
limits.bytes = cap;
let mut work = Budget::new(limits).unwrap();
let (index, _) = Index::parse(&skeleton(), &sources, &mut work).unwrap();
assert_eq!(work.usage.bytes, cap);
let usage = work.usage;
let numeric = work.numeric_units();
let names = BTreeMap::from([(global(0x100), "ff".into())]);
assert_eq!(
Plan::renames(&index, &names, &work).unwrap_err().message,
"Source8 index/output byte limit exceeded."
);
let composition = Composition::renames(&index, &names, &work).unwrap();
let batch = composition.original_edits(&sources).unwrap();
assert_eq!(batch.len(), 3); // declaration and two distinct call targets
let growth: usize = batch
.iter()
.map(|edit| edit.replacement().len() - edit.original().len())
.sum();
assert_eq!(growth, 3);
assert_eq!(
composition.plan.apply(&sources, &work).unwrap_err().message,
"Source8 index/output byte limit exceeded."
);
// Independent future ordinary/scalar producer witness on untouched display
// name bytes. This is size/offset arithmetic only: this typed-only harness
// has no ordinary collector, mixed ownership proof or accepted output.
let start = DEFINITIONS.find("\"f a b\"").unwrap() + 1;
let shrink = Location {
file: 11,
span: Span {
start,
end: start + "f a b".len(),
},
};
assert_eq!(
&sources[&shrink.file][shrink.span.start..shrink.span.end],
"f a b"
);
for edit in batch.iter() {
let at = edit.location();
assert!(
at.file != shrink.file
|| at.span.end <= shrink.span.start
|| shrink.span.end <= at.span.start
);
}
let original_len = shrink.span.end - shrink.span.start;
assert_eq!(cap + growth - original_len + "fa".len(), cap);
assert_eq!(cap + growth - original_len + "fab".len(), cap + 1);
assert_eq!(work.usage, usage);
assert_eq!(work.numeric_units(), numeric);
}
#[test]
fn composition_keeps_name_storage_and_rename_record_limits_before_allocation() {
let sources = sources();
let cap = sources.values().map(String::len).sum();
let mut limits = Limits::source8(ParseLimits::UNLIMITED);
limits.bytes = cap;
let mut work = Budget::new(limits).unwrap();
let (index, _) = Index::parse(&skeleton(), &sources, &mut work).unwrap();
let exact = BTreeMap::from([(global(0x100), "x".repeat(cap))]);
assert_eq!(
Composition::renames(&index, &exact, &work)
.unwrap()
.original_edits(&sources)
.unwrap()
.len(),
3
);
let over = BTreeMap::from([(global(0x100), "x".repeat(cap + 1))]);
assert_eq!(
Composition::renames(&index, &over, &work)
.unwrap_err()
.message,
"Source8 index/output byte limit exceeded."
);
for invalid in [
BTreeMap::from([(global(999), "unknown".into())]),
BTreeMap::from([(global(0x100), "bad name".into())]),
BTreeMap::from([(global(0x100), "g".into())]),
BTreeMap::from([(parameter(0x100, 0x200), "b".into())]),
] {
assert!(Composition::renames(&index, &invalid, &work).is_err());
}
// A typed-empty fixture isolates the rename-map record bound without
// syntax entry usage masking it. No ordinary edits are authorized here.
let ordinary = skeleton();
let count = ordinary.identities().len();
let files = BTreeMap::from([(1, "// empty typed unit\n".into())]);
let mut limits = Limits::source8(ParseLimits::UNLIMITED);
limits.entries = count;
limits.references = 0;
let mut inherited = Budget::new(limits).unwrap();
let (index, _) = Index::parse(&ordinary, &files, &mut inherited).unwrap();
let mut names: BTreeMap<_, _> = index
.names()
.keys()
.enumerate()
.map(|(i, target)| (*target, format!("renamed_{i}")))
.collect();
assert_eq!(names.len(), count);
assert!(
Composition::renames(&index, &names, &inherited)
.unwrap()
.original_edits(&files)
.unwrap()
.is_empty()
);
names.insert(global(999), "additional".into());
assert_eq!(
Composition::renames(&index, &names, &inherited)
.unwrap_err()
.message,
"Source8 lowering entry limit exceeded."
);
}
#[test]
fn composition_checks_edit_record_ceiling_before_duplicate_membership() {
let sources = sources();
let mut limits = Limits::source8(ParseLimits::UNLIMITED);
limits.entries = 128;
limits.references = 32;
let ceiling = limits.entries + limits.references;
let mut work = Budget::new(limits).unwrap();
let (index, _) = Index::parse(&skeleton(), &sources, &mut work).unwrap();
let mut composition = Composition::renames(&index, &rename_map(), &work).unwrap();
assert_eq!(composition.original_edits(&sources).unwrap().len(), 13);
let duplicate = composition.plan.edits[0].clone();
composition.plan.edits = vec![duplicate.clone(); ceiling];
// At the exact count bound validation proceeds to the independent ownership
// error. The damaged plan never becomes a successful/authoritative batch.
assert_eq!(
composition.original_edits(&sources).unwrap_err().message,
"Forged or duplicate edit occurrence."
);
composition.plan.edits.push(duplicate);
assert_eq!(
composition.original_edits(&sources).unwrap_err().message,
"Source8 lowering occurrence limit exceeded."
);
}
#[test]
fn standalone_noop_apply_keeps_tighter_file_only_budget_and_output_boundary() {
let sources = sources();
let mut inherited = budget();
let (index, _) = Index::parse(&skeleton(), &sources, &mut inherited).unwrap();
let plan = Plan::renames(
&index,
&BTreeMap::from([(global(0x100), "f".into())]),
&inherited,
)
.unwrap();
assert_eq!(plan.edit_count(), 0);
assert!(index.names().len() > sources.len());
let mut limits = Limits::source8(ParseLimits::UNLIMITED);
limits.entries = sources.len();
limits.references = 0;
limits.bytes = sources.values().map(String::len).sum();
// Standalone apply checks file/edit records, not the already validated
// catalog/name-storage allocation from construction. Rechecking the catalog
// under this intentionally tighter probe budget would reject this no-op.
assert_eq!(
plan.apply(&sources, &Budget::new(limits).unwrap()).unwrap(),
sources
);
limits.bytes -= 1;
assert_eq!(
plan.apply(&sources, &Budget::new(limits).unwrap())
.unwrap_err()
.message,
"Source8 index/output byte limit exceeded."
);
}
@@ -194,6 +194,8 @@ pub(super) fn row_key(source: &str, origins: &BTreeMap<Id, Id>) -> Option<String
cursor: 0,
value_depth: 0,
limits: ParseLimits::UNLIMITED,
#[cfg(test)]
mixed: None,
};
let ast = parser.value().ok()?;
let facet = serde_json::from_value(raw(&ast).ok()?).ok()?;
+7 -147
View File
@@ -8,6 +8,7 @@ use std::collections::{BTreeMap, BTreeSet};
mod source_migration;
#[path = "source_project/source_structure.rs"]
mod source_structure;
pub(crate) mod traversal;
/// A portable slash-separated path, checked at every bundle boundary.
pub type LogicalPath = String;
@@ -304,71 +305,7 @@ fn compile_once(
bundle: &SourceBundle,
limits: ProjectLimits,
) -> Result<(CompiledSourceProject, Edits), Vec<SourceDiagnostic>> {
let fail = |code, message| {
vec![problem(
bundle,
&bundle.entry,
Span::default(),
code,
message,
)]
};
if bundle.entry.contains('/') {
return Err(fail(
"invalid_path",
"The entry file must be at the source project root.".into(),
));
}
if bundle.files.len() > limits.max_files {
return Err(fail(
"project_limit",
format!("Source project exceeds {} files.", limits.max_files),
));
}
let bytes = bundle
.files
.values()
.fold(0usize, |sum, text| sum.saturating_add(text.len()));
if bytes > limits.model.max_source_bytes {
return Err(fail(
"source_limit",
format!(
"Combined source exceeds {} bytes.",
limits.model.max_source_bytes
),
));
}
let mut folded = BTreeMap::new();
for path in std::iter::once(&bundle.entry).chain(bundle.files.keys()) {
if resolve_include("", path).as_ref() != Ok(path) {
return Err(fail(
"invalid_path",
format!("Non-canonical source path '{path}'."),
));
}
if let Some(previous) = folded.insert(path.to_lowercase(), path)
&& previous != path
{
return Err(fail(
"duplicate_path",
format!("Case-colliding source paths '{previous}' and '{path}'."),
));
}
}
if !bundle.files.contains_key(&bundle.entry) {
return Err(fail(
"missing_include",
"The entry file is missing from the source bundle.".into(),
));
}
enum Visit {
Enter(LogicalPath, usize, Option<SourceLocation>),
Leave,
}
let mut stack = vec![Visit::Enter(bundle.entry.clone(), 0, None)];
let mut active: Vec<LogicalPath> = Vec::new();
let mut visited = BTreeSet::new();
let mut traversal = traversal::Traversal::new(bundle, limits)?;
let mut declarations = Vec::new();
let mut declaration_paths = Vec::new();
let mut aliases = BTreeMap::new();
@@ -381,51 +318,8 @@ fn compile_once(
let mut includes = Vec::new();
let mut insertions: Edits = BTreeMap::new();
let mut graph_count = 0usize;
while let Some(visit) = stack.pop() {
let Visit::Enter(path, depth, origin) = visit else {
active.pop();
continue;
};
let at = origin.unwrap_or(SourceLocation {
path: path.clone(),
span: Span::default(),
});
if let Some(start) = active.iter().position(|file| file == &path) {
let chain = active[start..]
.iter()
.chain(std::iter::once(&path))
.cloned()
.collect::<Vec<_>>()
.join(" -> ");
return Err(vec![problem(
bundle,
&at.path,
at.span,
"include_cycle",
format!("Include cycle: {chain}."),
)]);
}
if depth > limits.max_include_depth {
return Err(vec![problem(
bundle,
&at.path,
at.span,
"project_limit",
format!("Include depth exceeds {}.", limits.max_include_depth),
)]);
}
if !visited.insert(path.clone()) {
continue;
}
let Some(source) = bundle.files.get(&path) else {
return Err(vec![problem(
bundle,
&at.path,
at.span,
"missing_include",
format!("Included file '{path}' is missing."),
)]);
};
while let Some(path) = traversal.next()? {
let source = &bundle.files[&path];
let mut unit = language::parse_unit(source, path == bundle.entry, limits.model).map_err(
|diagnostic| {
vec![SourceDiagnostic {
@@ -513,46 +407,12 @@ fn compile_once(
declarations.push(declaration);
declaration_paths.push(path.clone());
}
let mut children = Vec::new();
for (raw, span) in unit.includes {
let target = resolve_include(&path, &raw)
.map_err(|message| vec![problem(bundle, &path, span, "invalid_path", message)])?;
let location = SourceLocation {
path: path.clone(),
span,
};
includes.push(IncludeLocation {
location: location.clone(),
target: target.clone(),
});
if includes.len() > limits.max_include_edges {
return Err(vec![problem(
bundle,
&path,
span,
"project_limit",
format!("Include graph exceeds {} edges.", limits.max_include_edges),
)]);
}
children.push(Visit::Enter(
target,
depth.saturating_add(1),
Some(location),
));
includes.push(traversal.include(&path, &raw, span)?);
}
active.push(path);
stack.push(Visit::Leave);
stack.extend(children.into_iter().rev());
}
if let Some(path) = bundle.files.keys().find(|path| !visited.contains(*path)) {
return Err(vec![problem(
bundle,
path,
Span::default(),
"unreachable_file",
"Supplied source file is not reachable from the entry file.",
)]);
traversal.finish_file(path);
}
traversal.finish()?;
let header = project_header.expect("validated entry header");
let index = SourceIndex {
declarations: declaration_locations,
@@ -0,0 +1,258 @@
//! Shared include traversal policy for ordinary compilation and private syntax.
use super::*;
enum Visit {
Enter(LogicalPath, usize, Option<SourceLocation>),
Leave,
}
pub(crate) struct Traversal<'a> {
bundle: &'a SourceBundle,
limits: ProjectLimits,
stack: Vec<Visit>,
active: Vec<LogicalPath>,
visited: BTreeSet<LogicalPath>,
children: Vec<Visit>,
depth: usize,
edges: usize,
}
impl<'a> Traversal<'a> {
pub(crate) fn new(
bundle: &'a SourceBundle,
limits: ProjectLimits,
) -> Result<Self, Vec<SourceDiagnostic>> {
let fail = |code, message| {
vec![problem(
bundle,
&bundle.entry,
Span::default(),
code,
message,
)]
};
if bundle.entry.contains('/') {
return Err(fail(
"invalid_path",
"The entry file must be at the source project root.".into(),
));
}
if bundle.files.len() > limits.max_files {
return Err(fail(
"project_limit",
format!("Source project exceeds {} files.", limits.max_files),
));
}
let bytes = bundle
.files
.values()
.fold(0usize, |sum, text| sum.saturating_add(text.len()));
if bytes > limits.model.max_source_bytes {
return Err(fail(
"source_limit",
format!(
"Combined source exceeds {} bytes.",
limits.model.max_source_bytes
),
));
}
let mut folded = BTreeMap::new();
for path in std::iter::once(&bundle.entry).chain(bundle.files.keys()) {
if resolve_include("", path).as_ref() != Ok(path) {
return Err(fail(
"invalid_path",
format!("Non-canonical source path '{path}'."),
));
}
if let Some(previous) = folded.insert(path.to_lowercase(), path)
&& previous != path
{
return Err(fail(
"duplicate_path",
format!("Case-colliding source paths '{previous}' and '{path}'."),
));
}
}
if !bundle.files.contains_key(&bundle.entry) {
return Err(fail(
"missing_include",
"The entry file is missing from the source bundle.".into(),
));
}
Ok(Self {
bundle,
limits,
stack: vec![Visit::Enter(bundle.entry.clone(), 0, None)],
active: Vec::new(),
visited: BTreeSet::new(),
children: Vec::new(),
depth: 0,
edges: 0,
})
}
pub(crate) fn next(&mut self) -> Result<Option<LogicalPath>, Vec<SourceDiagnostic>> {
while let Some(visit) = self.stack.pop() {
let Visit::Enter(path, depth, origin) = visit else {
self.active.pop();
continue;
};
let at = origin.unwrap_or(SourceLocation {
path: path.clone(),
span: Span::default(),
});
if let Some(start) = self.active.iter().position(|file| file == &path) {
let chain = self.active[start..]
.iter()
.chain(std::iter::once(&path))
.cloned()
.collect::<Vec<_>>()
.join(" -> ");
return Err(vec![problem(
self.bundle,
&at.path,
at.span,
"include_cycle",
format!("Include cycle: {chain}."),
)]);
}
if depth > self.limits.max_include_depth {
return Err(vec![problem(
self.bundle,
&at.path,
at.span,
"project_limit",
format!("Include depth exceeds {}.", self.limits.max_include_depth),
)]);
}
if !self.visited.insert(path.clone()) {
continue;
}
if !self.bundle.files.contains_key(&path) {
return Err(vec![problem(
self.bundle,
&at.path,
at.span,
"missing_include",
format!("Included file '{path}' is missing."),
)]);
}
self.depth = depth;
return Ok(Some(path));
}
Ok(None)
}
pub(crate) fn include(
&mut self,
path: &str,
raw: &str,
span: Span,
) -> Result<IncludeLocation, Vec<SourceDiagnostic>> {
let target = resolve_include(path, raw)
.map_err(|message| vec![problem(self.bundle, path, span, "invalid_path", message)])?;
let location = SourceLocation {
path: path.into(),
span,
};
if self.edges >= self.limits.max_include_edges {
return Err(vec![problem(
self.bundle,
path,
span,
"project_limit",
format!(
"Include graph exceeds {} edges.",
self.limits.max_include_edges
),
)]);
}
self.edges += 1;
self.children.push(Visit::Enter(
target.clone(),
self.depth.saturating_add(1),
Some(location.clone()),
));
Ok(IncludeLocation { location, target })
}
pub(crate) fn finish_file(&mut self, path: LogicalPath) {
self.active.push(path);
self.stack.push(Visit::Leave);
self.stack.extend(self.children.drain(..).rev());
}
pub(crate) fn finish(self) -> Result<(), Vec<SourceDiagnostic>> {
if let Some(path) = self
.bundle
.files
.keys()
.find(|path| !self.visited.contains(*path))
{
return Err(vec![problem(
self.bundle,
path,
Span::default(),
"unreachable_file",
"Supplied source file is not reachable from the entry file.",
)]);
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn ordinary_compile_once_keeps_include_order_and_cycle_source_location() {
let mut bundle = SourceBundle {
entry: "main.arcadia".into(),
files: BTreeMap::from([
("main.arcadia".into(), "arcadia 7; project p \"P\" id \"00000000-0000-0000-0000-000000000001\" { include \"dir/left.arcadia\"; include \"dir/right.arcadia\"; }".into()),
("dir/left.arcadia".into(), "include \"../shared.arcadia\";".into()),
("dir/right.arcadia".into(), "include \"../shared.arcadia\";".into()),
("shared.arcadia".into(), "// one shared ordinary fragment\n".into()),
]),
};
let original = bundle.clone();
// This is the existing initial ordinary pass, before schema migration's
// separate output/reparse. It makes no public source8 acceptance claim.
let (compiled, insertions) =
super::super::compile_once(&bundle, ProjectLimits::default()).unwrap();
assert!(insertions.is_empty());
assert_eq!(compiled.bundle, original);
assert_eq!(compiled.index.declarations.len(), 1);
assert_eq!(
compiled
.index
.includes
.iter()
.map(|i| (i.location.path.as_str(), i.target.as_str()))
.collect::<Vec<_>>(),
[
("main.arcadia", "dir/left.arcadia"),
("main.arcadia", "dir/right.arcadia"),
("dir/left.arcadia", "shared.arcadia"),
("dir/right.arcadia", "shared.arcadia"),
]
);
let cycle = "// 雪\r\ninclude \"main.arcadia\";";
bundle.files.insert("shared.arcadia".into(), cycle.into());
let before = bundle.clone();
let errors = super::super::compile_once(&bundle, ProjectLimits::default()).unwrap_err();
assert_eq!(errors[0].diagnostic.code, "include_cycle");
assert_eq!(errors[0].path, "shared.arcadia");
assert_eq!(errors[0].diagnostic.line, 2);
assert_eq!(
&cycle[errors[0].diagnostic.span.start..errors[0].diagnostic.span.end],
"include \"main.arcadia\";"
);
assert_eq!(
errors[0].diagnostic.message,
"Include cycle: main.arcadia -> dir/left.arcadia -> shared.arcadia -> main.arcadia."
);
assert_eq!(bundle, before);
}
}