Named syntax-text producer contract — proposal

2026-09-25. Design only, not accepted requirements or implementation. Source pin 8d52dab97c92cc31baae9f20d8c324b55818aa5d; paths below are repository-relative under crates/oa-core/src/. Builds/runtime/publication remain parent-owned. This refines the named text rows of NEXT-PRODUCER-HOOKS.md and the domain audit; it does not implement PendingRename verification or close every parser timing obligation.

Smallest useful next slice

Add an explicit private text-producing mode that checks owned syntax text before its producer allocates/copies it, plus the shared scanner's validation-only decoded String. Keep the current legacy and carried-phase-only entry points and checkpoints intact. Enable the stronger mode only through an internally controlled fixture route for now, bound to the actual carried bundle/guard and exact inherited policy. A legitimate zero C still uses the stronger rules; do not enable/disable by its numeric value. The later PendingRename transition can select this mode only under its own accepted ownership contract.

This is independently useful: a successful stronger-mode syntax witness proves its counted retained text existed only after checks against the still-live carried text. It does not yet prove producer timing for all entry slots, traversal scratch or model/validator allocations. It does not expose an accepted output or change public source8 dispatch.

Exact domain

C is existing carried live Slots, immutable provenance. Its text includes all owned output sources, canonical keys and entry spelling. It is not work Usage.
T is the same retained syntax String inventory currently counted by mixed/project/accounting.rs:233–352 and Walk::drain (:146–230): ordinary project/header aliases/names/description, include paths, declaration aliases/names, AST field keys/String/Word/Reference text, separate field_spans keys, projected record keys/reference names, and the typed declaration/parameter/reference/type/expression string inventory. Each simultaneously owned retained copy counts its own UTF-8 length. Static header-role names, source slices and numeric literal spans are not copied text.
W is only named temporary text selected in this slice: scanner validation result, outstanding quoted-decode reservation slack, consumed UUID spelling, and local copies made by these hooks before transfer/refund. On successful syntax return W=0 and charged retained T equals the original independent retained-text inventory.
Every proposed reservation checks C.text + T + W.text + delta <= Limits.bytes, with checked arithmetic. This storage use does not increment bytes/tokens/entries/references/numeric work Usage, shrink the Limits object or reinterpret source-byte work. Existing carried entry/occurrence phase checks remain unchanged.

This is a logical string-length metric, not capacity/RSS/all heap. Existing raw model construction/serde model conversion, architectural validator heap, exact-number limbs, allocator overhead and native stack remain outside this metric. The scanner's returned decoded String is included despite being created by serde; it is not covered by a generic serde exemption. Internal serde decode scratch/capacity is not represented separately by T/W here and must be stated as excluded implementation scratch, not claimed covered by one result-String reservation. Typed duplicate-name coordination sets and traversal/ordinary-open-stack/projection-walk scratch retain their previously documented separate bounds/limitations; they are not silently included or certified by this text slice. Newly introduced containers would require an explicit additional reservation/design decision; the proposed hook state needs only scalar counts.

Quoted text policy: select raw payload reservation with refund

For a complete JSON string token let P be its UTF-8 byte length between the quotes, and D the decoded String byte length. Valid JSON guarantees D<=P: raw UTF-8 bytes copy unchanged; each simple escape consumes two source bytes and yields one; a non-surrogate six-byte \uXXXX yields at most three UTF-8 bytes; a twelve-byte surrogate pair yields four. Empty strings have P=D=0. JSON does not normalize Unicode or expand entities. Invalid escapes/surrogates do not produce a successful value.

Before the existing decoder call reserve P. On successful decode, check D<=P and release P-D; move the D charge with the String. If it is validation-only or a consumed identity/historical UUID spelling, drop the value before releasing its remaining reservation. A failed decode drops the decoder result and releases its attempted reservation before reporting failure where the scanner continues classifying tokens. No witness escapes a parser failure; there is no partial successful ledger to repair afterward.

This conservative policy can refuse a string whose decoded payload alone fits. That is an explicit new-mode admission rule: e.g. available storage 5 cannot admit "\u0061" (P6,D1), while available6 can reserve and later retain only1. Existing entry points do not gain that refusal. Do not report rawpayload policy as exact decoded-fit acceptance or hide its new error timing. Use the quoted token span internally for reserve refusals, preserving existing safe original-file transport. Current ordinary_error/located parser transport may report a one-byte public diagnostic at its start; this text slice does not silently change that diagnostic convention.

Candidate	Benefit	Cost / reason not selected
Exact D pre-count by an allocation-free JSON string grammar	Escaped strings admit at exact decoded-fit storage	Duplicates escape/surrogate/invalid-input interpretation or adds a second content pass; must track the actual serde dialect and error precedence. More code and compatibility proof than this bounded slice warrants.
Count via serde visitor	Appears to reuse serde grammar	Visitor callbacks can occur after serde has built escaped-string scratch or produced an owned result; counting there is not a pre-growth check. Merely using a custom Visitor does not prove early reservation.
Reserve raw P before the existing decoder, reconcile D	Simple proved bound, one existing decode per current consumer, no numeric reparse or multiplier	Conservative refusal must be accepted explicitly; library scratch/capacity is outside the selected logical result-text metric. Recommended.

Do not add a second decoder or a third scanner-validation pass. If exact decoded-fit compatibility is required, that is a consequential different contract decision before implementation.

Concrete reusable seams

The natural owner is the already shared Budget, because raw lexing, typed Cursor and ordinary mixed State borrow that one object. A private optional scalar producer-text state can expose reserve, reconcile/refund and final retained-text observation. It must not copy full Limits/ProjectLimits or add an owner-sized enum to each witness: use Budget's existing exact limits and the existing carried authority. State growth and parser stack controls must be measured after implementation, given the prior debug-frame regression.

Use named methods conceptually equivalent to reserve_text(bytes, span), release_text(bytes) and quoted begin/finish transfer; exact signatures are implementation choices. Refund uses checked subtraction, not saturation. Reserve errors do not install new totals. Moves are neutral; copies reserve before creation. Keep retained and selected temporary subtotals distinguishable so W=0 and retained inventory equality are checkable. No arbitrary public setter, caller-supplied unrelated C or fresh policy reset is introduced.

Producer / actual source	Required operation and lifecycle
`raw_tokens.rs:107–130`, allocation at126	Inside the same scanner, after finding a closing quote but before existing serde String validation, reserve P. Let validation finish and drop its String, then refund P. An after-`next()` adapter is too late. Preserve unclosed-token behavior: no decoder and no decoded-result reservation if no close was found.
`analysis_source/lex.rs:19–52`	Keep source-byte charge and raw-token charge before scanner advancement, including trivia. Call the hooked scanner only in the stronger mode. Retained significant tokens remain under the existing token bound; this patch must not introduce another tokens.len entry charge.
`analysis_source/cursor.rs:67–95`	Reserve exact spelling length before name/reference `.to_owned`; quoted calls reserve P then reconcile to D. Reference-work charge/cursor advance/error order stays unchanged.
`analysis_source/declarations.rs:405–434` identity; `language.rs:1211–1237` ordinary optional_id and `:1322–1332` historical uuid	The quoted UUID String is not retained syntax text. Keep it charged through UUID validation/parsing, drop it, then release D on success. No second UUID or Exact parse. Invalid identity aborts as before unless the earlier named storage refusal occurs.
`language.rs:1195–1210,1256–1275,1368–1374`	Ordinary quoted, identifier, field key, Ast Word producers reserve before allocation. A hook at mixed State makes the legacy path a no-op. Account retained quoted literals even when stored inside `serde_json::Value::String`: this is AST text, not excluded model conversion.
`language.rs:1245` nested fields; `:1518–1525` declaration field/field_spans	Each actual retained key copy is separate. Initial key transfers to one final owner. Reserve exact key.len before clone; refund original local if it drops rather than entering field_spans. Avoid hidden key clones in a convenience helper. A possible move-after-duplicate-check optimization requires separate error/order review, not a new metric exemption.
`language.rs:1047–1090` Positions fields/reference clone	Add a narrow fallible copy hook at each existing owned key/reference clone, aggregated with the still-live AST text. Preserve the shared legacy API via its no-op wrapper. Atom projection does not copy quoted payload; HistoricalUuid copies an Id, not a String. No extra decoded-text charge for either.
Other ordinary transient spellings and overwritten values	Inventory every successful-return discard of a charged producer result and refund it after drop; retain actual copied objects only once. Unrelated existing coordination copies absent from T are explicitly outside this slice, not silently charged as retained.

A minimal scanner seam is an internally shared next_with_string_hook body with one scoped callback/event for reserve/refund around its existing validation call. Iterator::next invokes the same body with an infallible no-op hook, so public/legacy tokens and malformed-token handling remain unchanged. The mixed lexer callback mutably borrows the single Budget for the duration of next; no callback is stored, no second scanner traverses the source and no validation String escapes the scanner. One callback with reserve/release events avoids two simultaneous mutable borrows of Budget. Alternate concrete signatures are acceptable if they preserve this timing and shared scan body.

The rawscanner hook may reserve after lexical cursor advancement to the known closing quote; that is still before the named String producer. A refusal terminates the stronger parse with no token/witness, not a resumable partially advanced scanner API. Reserve errors win over subsequent invalid-escape errors only in stronger mode; this ordering is deliberate and testable.

Inspected replacement and discard cases

These are actual source facts, not an assumption that every successful String copy survives:

Project description (language.rs:1560–1579): header.description.is_some rejects the second field before syntax_entry, quoted decoding or assignment. The first valid assignment replaces only the initially empty String, which has logical length0. There is no successful overwritten nonempty project description to refund. header.take() on a rejected duplicate drops its existing header during whole-parse failure; no successful witness remains.
Typed description (analysis_source/declarations.rs:301–317): the borrowed-key seen set rejects a duplicate field before the field match/quoted decoder/description assignment. The Option starts None. Thus each successful typed description has exactly one retained String; no blanket replacement subtraction is justified.
Declaration-kind word (language.rs:1587–1589, still lexically in scope through declaration push at1631): its current to_owned String is not in final T. Prefer replacing only this needless copy with the original source slice self.current().text(self.source), passing that borrowed word to declaration_kind and the unchanged unknown-kind diagnostic. The current Lexeme is a copied span; the borrowed text belongs to the source, not mutable Parser state. Preserve the same kind lookup and error formatting/order. This removes an otherwise new W lifetime without inventing a metric exemption. If inspection/compiler lifetimes prevent this exact borrow, reserve exact word.len before its existing allocation and explicitly drop/refund at the same safe point; do not leave its bytes in final T.
Nested map key clone (language.rs:1242–1252): field() produces an original key; fields.insert(key.clone(), value) retains a second allocation; the local original drops at loop end. Mark the original key's existing T charge as W (transfer, no aggregate delta), reserve the clone into T before copying, insert it, then explicitly drop/refund the original W. On a successful iteration one key length survives. In the top-level declaration case (:1518–1525), the original instead moves into field_spans while the clone stays in fields: both remain T. Do not apply the nested refund to that two-owner case.
Duplicate field insertion: nested/top-level insertion may replace a previous key/value immediately before returning the existing duplicate-field error. There is no successful replacement continuation. Keep existing diagnostic behavior/order and drop the failed parse; do not add a complex subtree refund traversal solely for a witness that cannot escape. Any newly prior storage refusal belongs only to the explicitly stronger mode.
Every successful quoted UUID discard found in this parser: ordinary optional_id (language.rs:1211–1237), ordinary historical uuid Ast branch (:1322–1332), and typed Cursor::identity (analysis_source/declarations.rs:405–434). Each temporarily retains its decoded spelling through validation/Id parsing, then returns only an Id and source spans. Mark its decoded charge temporary, explicitly drop the String, and refund D. HistoricalUuid does not become a counted text reference during projection (language.rs:1090) or inventory (accounting.rs:146–170). Typed parameter identities use the same Cursor identity path. These are three distinct consuming sites sharing the same quote producer, not three extra decoder passes.
Excluded coordination clones remain explicit: typed parameter_names (declarations.rs:293), override target names (:610) and Pratt sequence.keys (syntax.rs:244) own duplicate-check copies that do not occur in final T. This retained-text prerequisite does not claim to meter those existing scratch sets. Any later inclusion must reserve both entries and copied text before insertion and refund at actual drop; applying a blanket name-producer charge to these clones without release would be wrong.

The minimal live state can be a few scalar counts owned once in Budget, with narrow transfer/refund methods; it need not wrap every String in a lifetime-carrying type or refactor parser witnesses. Its representation and new helper frames still require measurement against the repaired parser's demonstrated roughly13 KiB numeric-path headroom improvement. No design can spend that margin based solely on source intuition.

Once-only resolver transfer and reconciliation

At successful stronger syntax return retain scalar T with Budget/text mode and require no live temporary reservation. Do not charge T as Usage. In resolve, seed the existing ledger with C once, then transfer/reserve known T once along with the established U+M path. Change the existing bounded syntax walk's text action in this mode to checked independent summation/reconciliation, rather than another text reservation. Require the independently walked retained String inventory to equal producer T before a resolved witness escapes. Keep its existing lower-node/occurrence counting and already-accounted work frontier; do not introduce a second complete AST clone or unbounded count traversal.

Legacy and existing phase-only mode keep their current incremental text walk. Stronger-mode full T remains live during the walk, so work peaks can tighten compared with the old progressive reservation; this is honest simultaneous ownership, not a compatibility promise. Entry-slot producer timing is still not changed by this slice. C must not be rebased away or added twice. M's token-end metadata remains its single established component, not a new charge for every significant token and projection table independently.

Finite acceptance cases

Unchanged APIs: same raw scanner token stream/spans/classes and legacy parser errors/models/Usage under existing tests, including malformed strings, trivia, Unicode/CRLF and source8 phase timing. Existing carried-phase tests keep their exact prior caps/diagnostics. Stronger mode is explicit, including C0.
Bound cases: independently hand-count empty payload0/0; "a"1/1; raw "é"2/2; "\\"2/1; "\n"2/1; "\u0061"6/1; "\u96ea"6/3; "\uD83D\uDE00"12/4. Test exact P and P-minus-one headroom as hook arithmetic; do not mislabel arbitrary scalar baselines as real owner provenance.
Actual minimal path: independent critic derived existing empty-project E with C.text102: scanner UUID result peaks at102+36=138; grammar retains alias1+display name7 before its UUID decode, so the selected producer peak is102+1+7+36=146. With sufficient entries17, inherited byte146 should pass and145 should refuse that grammar UUID temporary, leaving final retained T8 (C+T110) on success. This is a proposed real-path oracle, not a run result; preserve current located-span conventions. A real retained bundle fixture with two long escaped names/descriptions (each P greater than UUID36) distinguishes scanner temporary release from cumulative retained T. Derive complete C and all earlier peaks independently before selecting pipeline caps. Exact new-mode pass/refusal must use one inherited profile from entry through resolve, never mutate the live budget.
Named temporaries: repeated scanner validation does not retain its decoded strings; identity spellings disappear from final T; ordinary and typed UUIDs do not add36 each to final syntax. Raw validation occurs before grammar decoding but their result strings do not overlap. Invalid escape and unpaired-surrogate cases refund attempted scanner text and emit no syntax witness.
Duplicate copies: the existing ordinary description "é" inventory T45 is a discriminating final oracle: header8 + alias/name2 + field_spans key11 + AST key11 + decodedé2 + projected key11. Atom payload is not copied into positions. This is a roomy final reconciliation test, not a C+45/C+44 producer-cap oracle: a preceding declaration UUID causes a temporary text peak46, so C+44 can fail earlier and must not be asserted to reach projected-key allocation. Use a separately hand-counted later-copy-dominant fixture to isolate that hook. Include nested record keys, explicit @reference clone, typed scoped names and included files; reconcile final producer T against the independent existing syntax inventory.
New refusal semantics: long escaped input with D fitting but P exceeding available storage refuses at the quoted token before decoder growth in stronger mode; ample storage yields the same whole Project/Usage as legacy. Test resource refusal versus malformed escape ordering explicitly. Do not promise exactdecoded acceptance.
Ownership/accounting: checked overflow and invalid refund fault injection are labeled arithmetic tests; moves do not charge again; copies do; successful W0; resolver transfers C/T once. Actual complete bundle identity and exact Limits/ProjectLimits remain guarded. No hidden owned output extraction or adopted result.
Regressions: unchanged deep/wide parser and 128 KiB controls, meaningful measured native-frame headroom, all old failure dispositions, and exact numeric work. This slice cannot claim universal allocation containment, solve remaining traversal/entry-producer timing, or substitute byte/length tests for eventual actual-output semantic equality.

Decisions requiring acceptance before code

Select rawpayload conservative admission versus exactdecoded-fit compatibility; explicitly adopt the returned scanner String into W while excluding internal decoder scratch from this logical metric; decide whether the new mode must remain distinct from existing phase-only carried entry; and accept the exact list of still-open entry/traversal/coordination timing obligations. The recommendation is the explicit stronger mode, raw P/refund D, named result-String metric and once-only reconciled transfer above. If the predecessor's broad producer requirement is interpreted as every parser temporary already being in scope, this partial text slice must remain an identified prerequisite—not a silent weakening or final output-verification acceptance.

21 KiB Raw Blame History Unescape Escape