View on GitHub

ANI-Gamut

Benchmarking Agent Reliability across the Gamut of Agent–Network Interface Abstractions

LLM agents increasingly act on live networks: they read state, change configuration, and verify the result. A first-order question is usually left implicit: at which level of abstraction should the agent operate?

The same change can be presented to an agent as a raw device shell, as a typed declarative service request, or as an edit to a reconciled source of truth. These are very different contracts between the agent and the network. ANI-Gamut makes that level the variable we vary, holding the task, the model, and the observation interface fixed, and reads off reliability, safety, recovery, and cost.

Paper (CNSM 2026) · Extended version (PDF) · arXiv · Analysis plan · Talk · Code

The interface-abstraction spectrum

A0 A1 A1+ A2 A3-T A3-R A4

Green: implemented in ANI-Gamut  ·  grey: conceptual.

A0
Raw device shell: an unbounded action space, no transactional semantics.
A1
Bounded CLI wrapper: a generic guardrail (allowlist, management-plane protection) plus snapshot and manual rollback.
A1+
A1 plus an ad-hoc anti-damage blocklist, used as an ablation against the structural levels.
A2
Standardized model-driven configuration (NETCONF/YANG): typed, with candidate datastores and confirmed commit.
A3-T
Typed transactional service intent: a workflow transaction manager applies the request with active validation and rollback.
A3-R
Reconciled source of truth: a versioned desired state with drift detection and reconciliation.
A4
Service intent combined with reconciliation; where commercial controllers already operate.

The playground

A realistic brownfield

A single, densely populated multi-service substrate where many coexisting services share resources, so a careless change can shadow a prefix, reuse an identifier, or trip a global knob. This is where collateral damage actually arises, and where toy playgrounds hide it.

The same task, at every level

One harness exposes an identical configuration task to the agent through a per-level adapter, on one emulated testbed. Released as open source so the community can reproduce, extend, and compare.

ANI-Gamut architecture: a network operator states a configuration request over the human-agent interface (HAI) to the NMAA system, which acts on a brownfield network through the agent-network interface (ANI), whose abstraction level A0 to A4 is the controlled variable.
The system under test (an NMAA system) sits between the human–agent interface (HAI) and the agent–network interface (ANI); only the ANI level is varied, while the request, the model, and the observation are held constant.

The metric

Safe configuration is a zero-tolerance property: an agent that completes its task but silently breaks an unrelated, already-deployed service has caused an outage, not a success. The headline metric is therefore not mean task completion but the rate of damage-free runs, estimated with tail-aware statistics as the substrate is stressed by rising state density and injected faults. The hypothesis under test is that raising the interface level shrinks the agent’s blast radius, a structural safety gain that ad-hoc guardrails approximate but do not match.

Paper and resources

Code

The playground (harness and per-level adapters, scenario generator, SRv6 substrate) is released as open source in the netgroup/ani-gamut repository, tag cnsm2026 (playground/: Apache-2.0; substrate/: GPL-3.0).

How to cite

@inproceedings{bracciale2026anigamut,
  author    = {Bracciale, Lorenzo and Loreti, Pierpaolo and Mayer, Andrea and Salsano, Stefano and Henderickx, Wim},
  title     = {{ANI-Gamut}: Benchmarking Agent Reliability across the Gamut of Agent--Network Interface Abstractions},
  booktitle = {International Conference on Network and Service Management (CNSM)},
  address   = {Alcal\'a de Henares, Spain},
  year      = {2026}
}

Acknowledgment

This work has received funding from the Italian MUR PRIN NEWTON project.