Hypotheses and analysis plan
This page records the hypotheses, the experimental design, the metrics and the analysis plan of ANI-Gamut, as stated in Section VI (“Experimental Methodology”) of the CNSM 2026 paper. The last two sections state which part of the plan the pilot reported in the paper actually executed (Section VII) and the confound it does not resolve.
1. Hypotheses
We test six directional hypotheses, some of which may be refuted; that is what makes this measurement rather than advocacy.
- H1 (safety). Higher-abstraction interfaces lower the rate of collateral damage relative to the A0/A1 baseline, at constant task and model.
- H2 (recovery). Higher levels return the network to a safe, correct state more often after a committed error.
- H3 (expressiveness trade-off). Functional success is not monotonic in abstraction, because very high levels can cap what is expressible on some tasks.
- H4 (cost). Higher levels reach a correct-and-safe state at lower cost and lower variance.
- H5 (model × level). The benefit of abstraction is larger for weaker models, the interface compensating for model capability.
- H6 (attribution). The reliability gain comes from transactional and recovery semantics entering at specific points, not from typed representation alone.
2. Design and controlled factors
Treatment and factors. The interface level is the treatment in a factorial design that crosses it with the task class, the model, and the agent mode, swept over two stress parameters: the state density of the brownfield and the substrate fault rate λ.
Task classes. A task class is a configuration requirement stated on the human–agent interface (HAI): a prompt template with zero or more attached files. The task classes span the service classes that populate the substrate; two classes that request the same reconfiguration in different words isolate the effect of expression from the effect of the situation.
Variants and pairing. For each cell the generator draws S random variants of the task class, each a concrete request (tenant, prefixes, target) placed in a freshly generated brownfield. The prompt text and the file structure are identical across a class’s variants, so what varies is the situation, not the wording. These S variants are the per-cell runs over which the statistics are estimated, and each variant, bundling the request, the brownfield, and the fault seed, is replayed across the levels, which makes the comparison paired.
Matched budget. Levels are compared under a matched agent budget. The execution budget is held constant across levels: same model, system prompt, agent mode, step and token caps, retry policy, and observation interface, so that only the action interface differs. The development budget is equalized by a single shared agent with no level-specific tuning beyond exposing each interface. Cost (tokens, messages, steps, wall-clock) is a dependent variable (H4): the question is not only whether a higher level is safer at equal budget, but whether it reaches a safe state at lower budget.
Implemented slice. The levels implemented are A0, A1 and A3-T, with A1+ as an ablation; A2 and A3-R stay conceptual and A4 is future work. The claim concerns the trend from the low-abstraction baseline to the high-abstraction interfaces, not a continuous monotone curve, and attribution is preserved through within-level ablations rather than through the absent A2/A3-R contrasts.
3. Metrics: a zero-tolerance safety outcome
Why zero tolerance. Moving a success rate from 95% to 97% leaves a configuration agent far from deployable. A deployable agent must reach a success probability indistinguishable from one and damage no pre-existing service at carrier-grade reliability, a collateral-damage probability on the order of 10−5. No current system is near this bar; task completion and the breadth of damage do not rank near-deployable systems, but chart the directions in which the knobs (level, model, agent mode, stress) move a system.
Damaged instance. The unit of damage is the service instance: one deployed service, such as a single SRv6 tunnel, grouped into service classes by type. An instance is damaged if any of its baseline invariants fails after the agent acts, counted once however many invariants it carries.
Per-run measures.
- the per-class damage fraction Bc over all pre-existing service classes (SRv6 east-west and north-south, firewall, routing, floating-IP), plus the global invariants (backbone reachability, forwarding, default-route removal, VRF integrity);
- the binary safety outcome: the damage vector B = (Bc) is null or not.
Primary safety statistic. The damage-free run rate P(B = 0) over a set of runs: since collateral damage must be zero, incidence matters more than magnitude.
Secondary measures.
- Magnitude, diagnostic only: the macro-average B̄ across classes (equal class weight) and the per-class vector when damage occurs; pooled micro-averages only as an operational footprint.
- Functional success Stc: the fraction of test cases the newly configured service passes (the completion leg, not the safety headline).
- Outcome label per run: clean success, dangerous success (works but damages), clean failure, destructive failure.
- Recovery (H2): the fraction of fault-hit runs that still end damage-free and correct.
Oracle. Observation is out-of-band, so success and safety are scored identically at every level. The oracle reuses NetAiBench’s invariant checks and the SRv6 ICMP-in-SRv6 probe, with the scenario generator co-emitting the per-instance invariants.
4. Statistical analysis plan
- Estimand. The primary outcome is binary, so per cell we estimate p̂(damage) = k/S with a Wilson or Clopper–Pearson interval, since the normal approximation fails near zero, which is exactly the A3-T regime.
- Sample size. It follows from the zero claim, not from comparing means: by the rule of three, zero damages in S runs give an upper 95% bound near 3/S, so asserting p(damage) < 1% for A3-T needs S on the order of a few hundred per cell.
- Paired test. Common fault seeds are replayed across levels, which makes the comparison paired and lets McNemar’s test work on the discordant pairs, where the same physical event damaged the low level but not the high one.
- Secondary continuous metrics (cost, time-to-safe-state, magnitude when damage occurs): confidence intervals and mixed-effects regression with task and model as random effects, including the model × level interaction (H5).
- Reporting. Success and cost per level with interval estimates; density and λ sweeps in the extended version.
Attribution (H6). Contrasts probe one mechanism at a time, because A3-T and A3-R advance different axes rather than forming one staircase.
- A1 bundles bounding (allowlist, out-of-scope-shell block, management protection) and recovery (snapshot and manual rollback). They are varied as a 2×2 factor at constant free-text representation, separating how much preventing buys from how much undoing buys.
- The A1-to-A3-T step isolates service-scope typed intent with one-shot workflow transactions.
- The way an interface enforces consistency is itself a sub-axis: an ad-hoc prohibition (the A1+ blocklist), predetermined domain-specific templates, standardized validation and commit (A2), and typed transactional intent (A3-T) approach the same guarantee by different routes, and are compared empirically instead of assuming the typed route wins.
5. What the pilot actually executed
The results in the paper (Section VII) are a pilot and cover only part of the plan above:
- Levels measured: A0, A1, A1+ (ablation) and A3-T. A2 and A3-R are conceptual; A4 is future work.
- Tasks: a benign onboarding task and a change-set on a live 25-site tenant (two adds, two moves, one remove).
- Run counts: n = 20 per level on the benign task, n = 6 seeds on the change-set, n = 4 per level under faults (λ = 0.3, packet loss and node isolation).
- Model: one model, held fixed across levels, with a matched budget (same model, prompt and step cap).
- Statistics: Wilson intervals and exact paired McNemar tests. The run counts are far below the few hundred per cell required by the rule of three. With a single model, the model × level interaction (H5) cannot be estimated; the 2×2 bounding/recovery factorial is not reported in the conference paper.
- Outcome on H1: the damage-free rate is 1.00 at every level, on both tasks, benign and under faults. The agents fail safe, abandoning the task instead of corrupting their neighbors, and the substrate keeps tenants in separate VRFs: the substrate’s isolation bounds the blast radius, so on this substrate collateral damage is not governed by abstraction. A scenario with shared mutable state not protected by tenant isolation is the natural next study.
6. Known confound
Part of A3-T’s advantage under faults is that its actuation is out-of-band from the data-plane disturbance, so while the actuation channel is the same for every level, the low levels also pay for having to observe a degraded data plane. Typed representation, transactional semantics, the out-of-band actuation channel and the engineering that went into the adapter vary together in the pilot and are not separated; the matched budget equalizes the agent’s allowance, not these mechanisms. The extended version treats the actuation channel as a controlled variable.