| Internet-Draft | NMAA Benchmarking Framework | October 2026 |
| Salsano, et al. | Expires 3 May 2027 | [Page] |
Large Language Model (LLM) based agents are being proposed to configure and operate production networks. Whether a given Network Management AI Agent (NMAA) system is safe to deploy depends not only on the model it uses, but on the interface through which it acts, the way it is orchestrated, the kind of request it receives, and the state of the network it acts upon. This document describes a framework for controlled benchmarking of NMAA systems. The system under test is delimited by two interfaces: the human-agent interface, through which an operator states a requirement, and the agent-network interface, through which the system acts on the network. The framework varies one factor at a time (the model, the abstraction level of the agent-network interface, the agent mode, the task class) while holding the others fixed, under explicit stress parameters (the density of pre-existing configuration state and the rate of injected faults), on a brownfield substrate where collateral damage to pre-existing services can actually occur. It scores runs with a binary, zero-tolerance safety outcome and treats statistics as part of the design. A companion document specifies the corresponding benchmarking methodology.¶
This note is to be removed before publishing as an RFC.¶
The latest revision of this draft can be found at https://netgroup.github.io/draft-nmaa-benchmarking-framework/draft-salsano-nmrg-nmaa-benchmarking-framework.html. Status information for this document may be found at https://datatracker.ietf.org/doc/draft-salsano-nmrg-nmaa-benchmarking-framework/.¶
Discussion of this document takes place on the Network Management Research Group mailing list (mailto:nmrg@irtf.org), which is archived at https://mailarchive.ietf.org/arch/browse/nmrg. Subscribe at https://www.ietf.org/mailman/listinfo/nmrg/.¶
Source for this draft and an issue tracker can be found at https://github.com/netgroup/draft-nmaa-benchmarking-framework.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 3 May 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document.¶
Network operations are moving from commands typed by a human operator to software agents, typically built around an LLM, that observe the running network, change it, and confirm the result. Most of the attention in this space has gone to the agent itself: its model, its prompting, its planning loop. Several architectures for agentic network management have been proposed [I-D.wmz-nmrg-agent-ndt-arch] [I-D.zhao-nmop-network-management-agent], and first evaluation frameworks exist [I-D.cui-nmrg-llm-benchmark] [I-D.contreras-bmwg-ai-agent-benchmarking].¶
An operator who must decide whether to deploy such a system faces a different set of questions. Which model should the system use for a given class of tasks? Through which interface should it be allowed to act: a device shell, a bounded command wrapper, model-driven configuration, a typed service request, or an edit to a reconciled source of truth? Does a planner-executor agent behave better than a ReAct loop? Does the answer change when the network is densely configured, or when faults occur while the agent acts? Each of these questions is, in experimental terms, the same experiment with a different treatment: the system under test is kept fixed except for one factor, and the effect of that factor on success, safety and cost is measured.¶
This document describes a framework that makes that structure explicit. Its elements are:¶
a reference model that delimits the system under test between the human-agent interface and the agent-network interface (Section 4);¶
a set of controlled factors and stress parameters, and an experimental design (factorial, matched budget, paired runs) that compares settings of one factor while holding the others fixed (Section 4, Section 5);¶
a brownfield substrate, densely populated with heterogeneous pre-existing services, together with a generator of reproducible initial states and the invariants that those services must keep satisfying (Section 6);¶
a binary, zero-tolerance safety outcome, the damage-free run, with the statistics needed to bound near-zero damage probabilities (Section 4.7).¶
The emphasis on a brownfield substrate follows from the failure mode that matters in operations. A configuration agent that completes its task but silently breaks an unrelated, already-deployed service has caused an outage. Collateral damage of this kind can only occur where pre-existing services share resources with the target of the change: the same devices, address space, identifier pools, or global settings. A benchmark that runs agents on a nearly empty network cannot observe it.¶
One of the controlled factors, the abstraction level of the agent-network interface, deserves particular attention because it is the factor that is most often fixed implicitly. In software engineering, the interface offered to an agent, and not only the model behind it, is known to govern how reliably the agent acts [SWE-AGENT]. Operators are already climbing an abstraction ladder, from device command line to model-driven configuration ([RFC6241], [RFC8342]), to service-intent orchestration and reconciled sources of truth. Section 5.1 describes this ladder as a spectrum of levels that can be varied as a factor.¶
The framework is an outgrowth of an experimental study and of its open-source implementation [ANIGAMUT] [ANIGAMUT-EXT] [ANIGAMUT-CODE], summarized in Appendix A. This document does not report that study; it describes the framework so that other implementations and other campaigns can be designed and compared within it. The procedures, metrics definitions, statistics and reporting requirements are specified in a companion benchmarking methodology document [METHODOLOGY], written in the style of the benchmarking tradition of [RFC1242] and [RFC2544].¶
This document is a product of individual contributors and is submitted for discussion in the Network Management Research Group. It is not an IETF product and is not a standard.¶
This document uses the following terms.¶
A software system that receives a configuration or operational requirement from a human operator, or from another system acting on the operator's behalf, and acts on a network to satisfy it, using one or more AI models (typically LLMs) to decide its actions. The term covers single agents and multi-agent systems.¶
The interface through which a requirement is given to the NMAA system: a natural-language request with zero or more attached files (for example, a configuration file, a topology description, a spreadsheet of sites).¶
The interface through which the NMAA system observes and changes the network. The term was introduced in [I-D.cui-nmrg-llm-benchmark], where a single ANI is defined; this document treats the ANI as a factor that can take different abstraction levels (Section 5.1).¶
The boundary that encloses the NMAA system between the HAI and the ANI. The model, the orchestration logic and the ANI adapter are inside; the network substrate, the generator, the fault injector and the observer are outside. The SUT boundary corresponds to the "Compound System" boundary of [I-D.gaikwad-llm-benchmarking-terminology].¶
A network substrate that, before the agent acts, already carries a dense set of heterogeneous services sharing devices and resources.¶
One deployed service (for example, one tunnel, one firewall policy set for a tenant), the unit of collateral damage.¶
A type of service (for example, SRv6 east-west tunnels, firewall policy, underlay routing, floating IP); a brownfield carries instances of several service classes.¶
A machine-checkable condition that a pre-existing service instance, or the network as a whole, satisfies before the agent acts and must keep satisfying afterwards.¶
The violation of at least one baseline invariant of a pre-existing service instance, or of a global invariant, after the agent has acted.¶
A run after which no baseline invariant is violated.¶
A family of requirements stated on the HAI by a fixed prompt template and file structure (for example, "onboard a new tenant site").¶
A concrete instance of a task class (tenant, prefixes, target nodes), placed in a freshly generated brownfield.¶
A property of the SUT whose settings are compared in a campaign (model, ANI level, agent mode, task class).¶
A property of the environment that is swept to test whether a difference between factor settings persists, grows or disappears (state density, fault rate).¶
One execution of the SUT on one task variant, with one setting of every factor and stress parameter, under a fixed budget.¶
A set of runs designed to compare settings of one controlled factor.¶
+--------------------+
| Human operator |
+---------+----------+
| requirement (prompt + files)
=====================|============================ HAI
SUT boundary v
+------------------------------------------------+
| NMAA system |
| model(s) + agent mode + ANI adapter |
+----------------------+-------------------------+
=======================|========================== ANI (level A0..A4)
v
+------------------------------------------------+
| Brownfield substrate |
| (pre-existing services, shared resources) |
+------------------------------------------------+
^ ^
| fault injection | out-of-band observation
+----+-----+ +-----+------------------+
| injector | | observer (invariants, |
+----------+ | task test cases) |
+------------------------+
The SUT is the NMAA system between the HAI and the ANI (Figure 1). The level of the ANI is an internal choice of the NMAA system, invisible to the operator: the same HAI request can be met by an agent acting through a raw shell or through a typed service interface. The substrate, the scenario generator, the fault injector and the observer are part of the test environment, outside the SUT.¶
A campaign compares settings of one controlled factor; the others are held fixed. The controlled factors are:¶
the model, or the set of models, used by the NMAA system;¶
the abstraction level of the ANI (Section 5.1);¶
the agent mode: single-turn generation, ReAct-style loop, planner-executor, with or without a recovery step;¶
the task class, including the form of the HAI request.¶
Two stress parameters are swept across campaigns, to test whether a difference between factor settings is robust:¶
A task class is defined on the HAI by a prompt template and a file structure. For each cell of the design, the scenario generator draws S variants of the task class; each variant fixes the concrete request and is placed in a freshly generated brownfield. The prompt text and file structure are identical across the variants of a class, so that what varies between runs is the situation and not the wording. Two task classes that request the same reconfiguration in different words can be used to separate the effect of expression from the effect of the situation.¶
Settings of a factor are compared under a matched budget. The execution budget (model, system prompt, agent mode, step and token caps, retry policy, observation interface) is held constant except for the factor under study. The development budget is matched as well: when the ANI level is varied, a single shared agent is used, with no level-specific tuning beyond exposing each interface. Cost (tokens, messages, steps, wall-clock time) is recorded as a dependent variable, so that a factor setting can be found to reach a safe state at lower cost, not only to be safer at equal cost.¶
Each task variant bundles the request, the generated brownfield and the fault seed, and is replayed across the settings of the factor under study. This makes the comparison paired: the same physical event (the same fault, at the same point of the same scenario) is presented to every setting. Faults are injected in the substrate, below the ANI, so that the stimulus is identical for every setting and the comparison measures how the effect propagates through each SUT.¶
Success and safety are measured by the test environment, not by the SUT. The observer checks the test cases of the requested service and the baseline invariants of every pre-existing service instance after the run, through a channel that does not depend on the ANI level, so that every setting is scored identically.¶
The primary safety outcome of a run is binary: the run is damage-free or it is not. The primary safety statistic of a cell is the damage-free run rate. The completion outcome is the functional success of the requested service. Each run also receives an outcome label: clean success, dangerous success (the service works but damage occurred), clean failure, destructive failure. Cost, recovery after a fault, and the magnitude of damage when it occurs are secondary outcomes.¶
Because the target regime is a damage probability near zero, the statistics are part of the design and not an afterthought: proportions are estimated with interval methods valid near zero (Wilson, Clopper-Pearson), the number of runs per cell is sized from the bound to be asserted (by the rule of three, zero damaged runs out of S give an approximate 95% upper bound of 3/S), and paired comparisons use tests on discordant pairs (McNemar). Definitions and procedures are given in [METHODOLOGY].¶
The ANI levels form a spectrum from raw device access to reconciled service automation (Table 1).¶
| Level | Interface | Key safety or transactional property |
|---|---|---|
| A0 | raw device shell | none; unbounded action space |
| A1 | bounded per-device command wrapper | generic guardrail (allowlist), snapshot and manual rollback |
| A2 | model-driven configuration (NETCONF/YANG) | device-level candidate datastore, validation, confirmed commit |
| A3-T | typed transactional service intent | cross-device workflow transaction, active validation, rollback |
| A3-R | reconciled source of truth | versioned desired state, drift detection and reconciliation |
| A4 | service intent and source of truth combined | both properties of A3-T and A3-R |
A0 to A2 form an ordered base, ordered by how far the interface moves the agent from raw mutation and how much safety it interposes. A2 corresponds to standardized model-driven configuration with candidate datastores and confirmed commit [RFC6241] [RFC8342] [RFC8040].¶
Above A2 the levels form a plane spanned by two axes: scope, from per-device configuration to cross-device service intent, and state and control model, from one-shot actions to a persistent, versioned desired state continuously reconciled against the network. A3-T advances the first axis, A3-R the second; they are not ordered with respect to each other. A4 combines both, and is where several commercial service orchestrators operate.¶
A level denotes the abstraction exposed to the agent, not the stack used to enforce it, and a level need not be built on the one below. A typed service intent (A3-T) can be realized by serializing the intent into device commands, without A2. This decoupling is what allows the same task to be exposed at several levels on one substrate.¶
A level can be complemented by ad-hoc measures that do not change its abstraction, such as a blocklist of known destructive commands added to A1. Such measures are treated as ablations within a level, so that the contribution of a structural interface and of hand-tuned prohibitions can be separated.¶
The model factor covers the choice of model, model size and version, and inference settings. A campaign that fixes the ANI level and varies the model answers the question of which model is adequate for a task class at a given level; one that varies both can test whether a higher level compensates for a weaker model.¶
TODO: reporting of model identity and version for closed models; handling of model updates during a campaign.¶
TODO: single-turn, ReAct, planner-executor, multi-agent; with and without recovery; what is held fixed when the agent mode is the factor.¶
TODO: task taxonomy (onboarding, change-set on a live tenant, decommissioning, troubleshooting); HAI forms from terse ticket to structured intent; held-out targets placed to interact with existing state.¶
This section outlines the requirements on the test environment; details are left to future revisions.¶
Multi-service substrate. Instances of several service classes coexist on the same devices and share namespaces, address space, identifier pools and global settings, so that resource contention, and therefore collateral damage, is possible. Topology size and state density are independent scales: density is configuration state, not additional nodes, and can be swept on a modest topology.¶
Layered, seeded scenario generator. The initial state is described by a manifest that stacks service-class layers on a topology base. Each layer establishes its prerequisites, generates its instances as real device state together with their baseline invariants (so that the oracle is emitted with the state), and verifies them. The manifest and a seed determine the scenario. Every pre-existing service passes its invariants before the run.¶
Realism of the initial state. Service sizes and node load follow skewed distributions; shared-resource dependencies and prefix adjacency or overlap occur at a tunable rate, so that shadowing and collisions are genuine traps; the target of the task is placed to interact with existing state.¶
Substrate-level fault injection. Faults are injected at node and link level (loss, latency, link down, frozen node, daemon restart, held lock, concurrent writer, transient pool exhaustion), identically for every setting, never inside an ANI adapter. Partial application, double application on retry and lockout are measured outcomes, not injected stimuli.¶
TODO: minimum requirements for a conforming substrate; how the generator's realism can be assessed.¶
This section illustrates how operational questions map onto campaigns of the framework.¶
Question: through which interface should an agent be allowed to act? The model, agent mode, task class and budget are fixed; the ANI level is the treatment. A pilot of this campaign is reported in [ANIGAMUT], on a multi-tenant SRv6 brownfield of about twenty-five nodes with on the order of ten service classes. On a change-set task on a live 25-site tenant (two additions, two moves, one removal), an agent acting at A0 completed 0 of 6 paired variants and the same agent at A3-T completed 6 of 6 (exact McNemar test on the six discordant pairs, p = 0.031), using about 3 steps and 42k tokens per run against 40 steps (the cap) and about 375k tokens. Under injected faults (lambda = 0.3) on an onboarding task, A3-T completed 4 of 4 runs and A0 none. The damage-free run rate was 1.00 at every level: the agents failed safe and the substrate kept tenants in separate VRFs, so collateral safety was determined by the isolation of shared state rather than by the ANI level. This illustrates why the substrate must contain services whose shared state isolation does not protect, if collateral damage is to be measured at all.¶
Question: which model is adequate for a task class at a given level? The ANI level, agent mode, task class and budget are fixed; the model is the treatment. The paired design is unchanged: each task variant is replayed across models.¶
Question: does the orchestration of the agent matter more than the model or the interface? The agent mode is the treatment, with model and ANI level fixed.¶
Any of the campaigns above can be repeated across values of the state density and of the fault rate. A difference between settings that grows with density or with the fault rate indicates a property that small, sparse testbeds would hide.¶
TODO: how an operator can use results to place an agent at a level for a given service type; limits of emulated substrates; transfer of results to production; cost of campaigns.¶
Benchmarking NMAA systems requires granting agents the ability to change network configuration. Campaigns need to be run on isolated substrates that cannot reach production networks or management systems; this applies in particular to the lowest ANI levels, which give the agent an unbounded action space.¶
Results of a campaign characterize a SUT on a given substrate and budget. They do not establish that a system is safe to deploy; a damage-free run rate of 1.00 over S runs only bounds the damage probability, approximately, below 3/S at 95% confidence.¶
The framework measures accidental collateral damage by agents acting in good faith. Adversarial inputs (prompt injection through the HAI or through data read from the network, compromised tools) are out of scope of this version and are addressed by other work. Generic guardrails and blocklists can make a level appear safe by tuning to the benchmark; the framework treats them as explicit ablations so that this effect can be measured.¶
This document has no IANA actions.¶
ANI-Gamut [ANIGAMUT] [ANIGAMUT-EXT] is an open-source implementation of the framework [ANIGAMUT-CODE] (release tag cnsm2026). It provides:¶
a harness with per-level ANI adapters for A0, A1 and A3-T, A1 with an optional blocklist as an ablation, A2 and A3-R being defined but not yet implemented;¶
a multi-service SRv6 substrate on Containerlab with Linux and FRR nodes, using a NEXT-CSID data plane, with firewall, underlay routing, floating-IP and addressing services coexisting on the same nodes;¶
a layered, seeded scenario generator with co-emitted invariants;¶
substrate-level fault injection and out-of-band observation.¶
It builds on the benchmarking playground described in [PLAYGROUND]. A project page is available at https://netgroup.github.io/ani-gamut/.¶
The framework originates from work carried out with Lorenzo Bracciale, Pierpaolo Loreti and Andrea Mayer. The authors thank Frank Loh and Papasega Wade for discussions.¶