DEV Community: NW Field Memo

TACACS+ Failover Testing: Rejection, Outage, and Recovery Are Different Tests

NW Field Memo — Sun, 13 Sep 2026 13:18:45 +0000

A TACACS+ rollout needs more than a successful login. Test what happens when one server is unavailable, when none can answer, and when the service returns. For each state, check who authenticated the user, what the user could do, and what evidence was recorded.

That last part is easy to miss. “I got a prompt” does not tell us which server answered—or whether a local account was used instead.

I'm Goda, a network engineer in Japan. This article develops the outage-testing questions from my Japanese TACACS+ material. The examples below are illustrative test designs, not reported results from a customer environment.

The focus is device administration through an SSH CLI. Product behavior depends on the device, software release, AAA configuration, and management path.

Start with an expected outcome for each failure state

Before interrupting anything, write down the behavior your design requires.

The following is an example policy, not a universal TACACS+ requirement. In this example, ordinary users rely on centralized authentication and a separate emergency account is available through a deliberately configured local recovery path.

Test condition	Expected result in this example	Evidence to collect
Both servers available; valid ordinary account	Normal centralized access with the intended permissions	Device response, server that handled the request, authorization and accounting evidence
First server unavailable; second available	Access through the remaining server, within the agreed time limit	Failure condition, elapsed time, request handled by the remaining server, resulting permissions
All configured servers unreachable; ordinary account	No unintended local access	Device response and evidence explaining the authentication outcome
All configured servers unreachable; designated emergency account	The approved recovery path works with only its intended permissions	Local authentication evidence, role or privilege, recovery-operation results, available audit evidence
Central service restored	New sessions use the intended normal path and normal controls work again	Authentication, permitted and prohibited operations, accounting, removal of temporary test changes

If your design intentionally has no local fallback, do not add one just to make the fourth row pass. Define how authorized recovery is supposed to work and test that arrangement.

Also, “first server” means the server the device is expected to try under the recorded starting conditions. Do not assume that every implementation always selects a server with the label “primary.”

A rejection is different from a server that cannot answer

Do not simulate a server outage by entering the wrong password.

A server can be fully reachable and deliberately reject authentication. A timeout means no usable response arrived within the relevant limit. A protocol error is another condition again.

RFC 8907 distinguishes a completed negative decision from an incomplete exchange: a FAIL result is applied as a decision, while an ERROR must be treated as though the server could not be reached. Available alternative methods may then be used. The details of redundancy, fallback, and timeouts are implementation-specific. RFC 8907, section 4.4

That gives us separate test items:

A validly formed login request that receives an explicit authentication rejection.
A request whose intended server does not respond.
An error condition, if that condition is part of the test scope.

For each one, record what actually happened. Do not label every failed login “server down,” and do not assume every failure should move on to a local account.

Check the target platform's documented method-list behavior and your configured policy before writing the expected result.

A one-server outage should prove which server took over

Suppose the design has two TACACS+ servers and access should continue when either one is unavailable.

“I disconnected one server and could still log in” is incomplete evidence. The device might have used the other server, a local account, or an existing session that never performed a new login.

A useful test sequence is:

Record the starting configuration and confirm normal access.
Establish the approved failure condition for one server.
Start a new management session with the intended test account.
Identify which authentication source handled that attempt.
Check the resulting permissions and relevant records.
Record the elapsed time and compare it with the acceptance limit chosen before the test.
Restore the starting conditions before testing the opposite direction.

Choose the failure mechanism carefully. Dropping traffic, rejecting a connection, and stopping a service can produce different observations. Record which condition you created instead of calling all of them the same outage.

Existing sessions can be a separate test item. Whether an already-open session can run its next command may depend on a different authorization exchange.

Local login is only the first part of emergency access

An emergency account is useful only if it supports the recovery work it was designed for.

Check these questions separately:

Did local authentication succeed?
Was a management session actually established?
Which role or privilege was applied?
Could the account perform the explicitly approved recovery operations?
Were operations outside that scope controlled as designed?
What record of the activity remained available?

TACACS+ treats authentication, authorization, and accounting as separate functions. Passing one stage does not establish the outcome of the others. RFC 8907, section 1

There is a concrete platform example in the Nexus 9000 NX-OS 10.4(x) guide. Command authorization has its own fallback configuration. Local command authorization is used only when the configured server groups fail to respond and local fallback has been configured; without that fallback, authorization fails. Console command authorization is configured separately as well. Cisco NX-OS 10.4(x): Configuring TACACS+

This is why a locally accepted password can still leave you unable to do the intended recovery work. Use the documentation for the actual product rather than copying NX-OS behavior to another platform.

Decide how to test safely before creating the outage

Use a suitable lab or an explicitly approved maintenance procedure. Establish an independent recovery path, define stop conditions, and prepare restoration steps before changing connectivity or AAA behavior.

A session that is already open is not automatically a sufficient recovery path: its next operation may still depend on the service you are about to interrupt.

Prefer harmless operations that demonstrate the required permission boundary. Do not try a restart, deletion, or disruptive configuration change on production equipment merely because the account is supposed to be denied.

For each disruptive test, identify:

What will be changed or interrupted.
What could be affected if the assumption is wrong.
How recovery can be performed if the tested management path fails.
When to stop, who restores the environment, and what evidence to preserve.

If those conditions are missing, record the item as not performed and state what is needed to run it.

“The server responds again” is not a complete recovery test

Restoring connectivity is a step toward recovery. It does not prove that normal administration has returned.

My suggested recovery checks are:

New authentication: Start a new session and identify the source that authenticates it.
Permissions: Repeat a representative permitted operation and a safely designed prohibited-operation test.
Accounting: Confirm that the expected new activity reaches the intended records.
Emergency access: Check that its post-recovery behavior matches the design. This does not necessarily mean deleting a deliberately retained emergency account.
Cleanup: Remove temporary failure-injection settings and verify the resulting configuration.
Handover: Record remaining gaps and the evidence that supports closure.

Do not require immediate return to a preferred server unless the product and configuration actually promise it. Server reactivation follows platform-specific policies and timers. For example, ASA 9.16 documents different reactivation modes, while NX-OS documents dead-time and probing behavior. Record the chosen settings and observed server selection. Cisco ASA 9.16: TACACS+ Servers for AAA

For ISE, use the report appropriate to the event. Its 3.5 guide lists separate TACACS Authentication, Authorization, Accounting, and Command Accounting reports. A login record does not substitute for command-accounting evidence. Cisco ISE 3.5: Device Administration

Also distinguish new records after restoration from records of activity during the outage. Seeing new records does not prove that every missing event was buffered and resent. If outage-period audit coverage is required, verify that behavior separately or document the approved alternative evidence.

Keep the failed run when the retest passes

Here is a fictional example:

First run: The emergency account passes local authentication, but a required recovery operation fails because its command authorization still depends on the unavailable service.

Judgment: Fail for the emergency-operation requirement. Keep the authentication success as an observation.

Next action: Review and correct the approved recovery design, then repeat the relevant test.

Retest: Add a new result linked to the original failure and the correction. Do not replace the failed row with “Pass.”

This makes the conclusion reviewable: someone can see the original condition, the issue, what changed, and what the next attempt demonstrated.

The same applies when evidence is incomplete. A test with an unexplained error is inconclusive, not automatically a successful denial.

The questions I want the record to answer are simple: Who authenticated the user? What could they do? What was recorded? What proves normal operation has returned?

Optional worksheets: free sample and paid kit

If you want an editable starting point, I publish two resources on note. Both the worksheets and their accompanying material are in Japanese. These are my own products.

Free sample: five basic TACACS+ administration checks.
Standard kit: 24 tests covering normal access, failure conditions, emergency access, and recovery. It is a one-time ¥800 purchase at the time of writing, with Excel worksheets, worked examples, a PDF guide, and fictional evidence examples. Purchase and download take place on note.

The kit is a test-design template, not a set of validated device configurations. Adapt the conditions, operations, and acceptance criteria to your equipment. Check the product page for the current price, contents, and usage terms.

For the permission checks discussed here, see my earlier article: Read-Only Access Testing: A Successful Login Is Not Enough.

Prepared with AI-assisted translation and editing of my Japanese material, with the additional protocol and product explanations checked against the official sources linked above.

Read-Only Access Testing: A Successful Login Is Not Enough

NW Field Memo — Sun, 13 Sep 2026 12:59:25 +0000

A read-only account needs to pass three separate checks: it can log in, it can read the information required for the job, and it cannot make the changes your requirements prohibit.

A successful login proves only part of that.

Adapted from my original Japanese article, with AI-assisted translation and editing.

I'm Goda, a network engineer in Japan. My Japanese blog has recently drifted into TFT and experimenting with AI, so this is also an attempt to remember what my job is.

In the previous article, I covered what to check when Cisco ISE Command Sets do not work as expected.

This time, the question is what we should put in a test plan before signing off on read-only access.

1. Turn “read-only” into three testable requirements

You sign in with the account. The management screen opens. You write “Pass.”

So far, so straightforward.

But can that account read the logs needed to investigate an incident? And can it really not change the settings it is supposed to leave alone?

Check	Example acceptance criterion
Can log in	A management session starts using the intended authentication source, user, and permissions.
Can read what is needed	The account can view the required status and logs for the resources it is allowed to access.
Cannot make prohibited changes	A valid prohibited operation is rejected by the applicable permission control, and the target remains unchanged.

These are examples for designing tests, not results from a device test. Choose actual operations and acceptance criteria for your product and requirements.

“Nothing could be changed” is not sufficient if the account also cannot read the logs that its operator needs. Likewise, being able to read logs does not establish that changes are blocked.

TACACS+ separates authentication, authorization, and accounting. A successful authentication record therefore cannot, by itself, establish which subsequent operations were permitted. RFC 8907, section 1

2. Define what the account may read and what it must not do

“It is a read-only user. Should I just try some show commands?”

First, define the job the account is intended to support.

Before testing, decide:

Which devices or resources are in scope.
Which status, logs, and configuration information the operator needs.
Which information or resource scopes must remain inaccessible.
Which operations are prohibited, such as configuration changes, saving changes, restarting, or user administration.
Which management interfaces are actually provided: CLI, GUI, or API.

Read-only does not necessarily mean permission to read every piece of information. A read operation can still expose sensitive information or impose processing load.

Translate the requirement into operations before deriving tests from the role name.

“Can view the status and necessary logs of the assigned resources” gives us a more useful scope than “has read-only access.”

3. Do not try a dangerous operation because it “should be denied”

This needs to come before the negative tests:

For tests involving configuration changes, saving, restarting, or other potentially disruptive operations, establish the impact and recovery procedure if the operation succeeds. Use a suitable test environment. Do not try a dangerous operation on a production device on the assumption that it will be rejected.

We are testing whether the permission control is correct. If it is wrong, the action may go through.

If the device restarts at that moment, we will certainly have a test result. We will also have another problem.

Before testing a change, identify the test resource, possible impact, recovery method, and stop conditions. If an operation cannot be tested safely, record why it was not performed and what remains unverified.

4. An error message does not automatically mean “Pass”

You attempt a prohibited operation. An error appears.

That looks promising. But what if you simply mistyped the command?

A syntax error does not prove that authorization would reject a valid operation.

The following are illustrative interpretations, not actual test results:

Observation	What the evidence supports
A valid operation is rejected for insufficient permissions, and the target is unchanged	Denial was verified for that operation under those test conditions.
The input produces a syntax error	The attempted operation was invalid. Permission-based denial remains unverified.
The connection drops before execution	There is a connection or processing problem. This is not evidence of authorization denial.
A change button is hidden	That screen hides the button. Other management paths remain unverified.
Authorization permits the operation, but execution fails for another reason	The operation was not stopped by the permission control.

If button visibility is itself a requirement, testing it can satisfy that requirement. It does not establish that the same change is impossible through another interface.

Separate what failed from why it failed.

Use an operation valid for the product, identify the reason for rejection, and check the target's state. “An error appeared” is too broad an acceptance criterion.

5. “Cannot deploy” and “cannot edit” are different restrictions

Cisco NDFC provides a useful documented example.

In the NDFC 12.2.2/12.2.3 guide:

Network Operator can view configuration information but cannot change expected switch configurations or deploy them.
Network Stager can prepare configuration changes inside NDFC but cannot deploy them to switches.

The Stager deployment restriction includes deployment-related Web UI and REST API actions. These are NDFC application roles; do not assume that a similarly named device role behaves identically. Cisco NDFC role documentation

A result of “could not deploy to the switch” does not tell us whether the user could edit configuration intent inside the management system.

Editing in the interface, saving in the management system, and applying to a device may be separate operations. Test the stages your requirements prohibit.

This example is scoped to the documented NDFC versions. Do not transfer it unchanged to another product or release.

6. Connect the operation, permission decision, and actual outcome

“Changes blocked: OK” is not a useful record when someone later asks what happened.

Which operation? Which user? Did a permission check reject it? Was the target actually unchanged?

Keep these pieces of evidence connected:

Target, management interface, user, and applied permissions.
Exact operation and arguments, execution time, and device or UI response.
The corresponding authorization decision, or evidence of the device's own permission enforcement.
The target-state check and the location of the supporting evidence.

If the device requests per-command authorization from ISE, trace the corresponding request and authorization result.

If the device enforces an assigned role locally, check the assigned role and the device's execution results and audit records.

ISE provides separate TACACS authentication, authorization, accounting, and command-accounting reports. Choose records appropriate to the control method and the accounting actually configured. Cisco ISE 3.5: Monitor Device Administration Activity

Do not make “every operation appears in ISE command authorization logs” a universal acceptance criterion for every product. Do not assume that every denied operation produces a command-accounting record either.

Correlate the target and operation as well as the time and username.

Here is a fictional example of a test record:

Target: Lab resource A, accessed through the GUI with the read-only test account.

Operation: Attempt a valid change to a setting that the requirements prohibit this account from changing.

Result: Rejected for insufficient permissions. Verified that the setting remained unchanged.

Evidence: The operation's UI response, the corresponding audit record, and the before/after setting checks.

Scope of conclusion: This user, resource, interface, and operation. CLI and API checks are separate test items.

For an actual record, fill in the timestamp, evidence IDs, exact setting, and other identifying details required by the test plan.

7. Keep untested scope out of the pass result

A CLI test is not a GUI test or an API test.

For each management path that is provided and used, verify the necessary read operations and the prohibited actions. You do not need to enable an unused management interface just to test it.

If permissions have recently changed, record the session state too. Whether a permission change affects an existing session, and when, depends on the product. Keep those results distinct from tests using a new session.

Finally, do not turn every empty cell into “OK.”

Not performed: Record the reason and the conditions needed to run it.
Not applicable: Explain why the item is outside the relevant scope.
Inconclusive: Record the missing evidence needed for a decision.

I want the test plan to show how far the verification actually goes.

Can log in. Can read what is needed. Cannot make prohibited changes.

Separating those three questions gives us something concrete to write after “Read-only login: successful.”

A sample worksheet

I have published a free TACACS+ test checklist sample in Japanese for turning these ideas into test items.

It covers five basic CLI administration checks. It is a test-design template that has not been validated on actual devices, not a ready-made certification of your environment. Adapt the operations, acceptance criteria, and evidence to the target equipment. GUI and API tests need separate design.

For troubleshooting restrictions that are not behaving as intended, see Cisco ISE Command Sets Not Working? Check Authorization Before Your Deny Rules.

The examples in this article were created for explanation and are not reported results from tests performed on actual equipment. Product-specific statements are limited to the versions identified in the linked documentation.

Cisco ISE Command Sets Not Working? Check Authorization Before Your Deny Rules

NW Field Memo — Mon, 07 Sep 2026 13:00:00 +0000

Boss: “No overtime. (But finish the work.)”

Me, a lowly network engineer: “Understood. (Overtime wouldn't be enough anyway.)”

Hi, I'm Goda, the network engineer behind NW Field Memo.

When you work with TACACS+ authorization in Cisco ISE, you come across Command Sets. Put simply, they let you define which commands to allow and which to deny.

My first impression was straightforward:

Create a Command Set for a read-only user → allow the required show commands → select it in the authorization policy → read-only access, done.

It turned out to be less straightforward.

Creating a Command Set does not prove that an operation is being controlled by it.

Here is the order I use to investigate when an ISE Command Set does not restrict an operation as expected.

What does a Command Set actually do?

A Command Set is an authorization result used in ISE Device Administration. It defines permit/deny decisions for commands.

An illustrative policy might allow show and ping, while denying configure and reload. This is a conceptual example, not a complete read-only policy or a configuration to copy into production.

The key dependency is the network device: it must request command authorization from ISE for the operation you want to control.

It is tempting to jump straight to the ISE configuration screen. I certainly did. But first, separate two questions.

“Can I log in?” and “Can I run this command?” are different questions

TACACS+ separates the three parts of AAA:

Function	The question it answers
Authentication	Who are you?
Authorization	What are you allowed to do?
Accounting	What did you do?

A successful login with an ISE user does not demonstrate that Command Set authorization works.

A simplified authentication flow looks like this:

User attempts to log in
  → Network device asks ISE to authenticate the user
  → Authentication succeeds
  → User logs in

Command authorization involves another decision:

User enters a command, such as show running-config
  → Network device asks ISE whether that command is allowed
  → ISE returns a permit/deny decision
  → Network device handles the operation

These are simplified flows, not packet traces. The point is that the first successful exchange does not prove the second exchange happened. RFC 8907 defines authentication, authorization, and accounting separately.

Make “the Command Set doesn't work” more specific

These observations are different:

A prohibited CLI command ran.
A user could change a setting in the GUI.
A configuration page opened.
A change button could be clicked.
The change was actually saved or applied.

Start by recording what the user did, what should have happened, and what actually happened.

For example:

A read-only user logged in through the CLI. The user entered configure terminal. It was expected to be denied, but the device accepted it.

That gives you something concrete to trace. Also record the product, OS version, and access method.

Before editing the Command Set, find the authorization request

This is the main point of this article.

When a prohibited command runs, it is tempting to think, “Maybe I wrote the Deny rule incorrectly,” and immediately edit the Command Set.

Before doing that, ask:

Did ISE receive a command authorization request for that operation?

If the operation is not being submitted to ISE for command authorization, rewriting its Command Set cannot control that operation.

A useful sequence is:

Identify the exact operation being tested.
Find its corresponding authorization request in ISE.
Check which authorization policy rule matched.
Check which Command Sets were selected.
Then inspect the Command Set rules.

When reproducing an operation, use an authorized test environment and consider what happens if the operation succeeds. The read-only testing section below explains why this matters.

“I cannot find a request” does not immediately mean “unsupported”

An authorization record may be missing from the logs you are looking at. That alone does not prove that the product lacks command authorization support.

Check:

Whether command authorization is configured on the device.
Whether you are tracing the same user and session.
Whether the log type, time range, and filters are correct.
Whether that access method and operation use command authorization.

“I could not find the log entry” is an observation.

“This product does not support that function” is a conclusion that requires further investigation.

Keeping those separate also makes a TAC case much easier to write.

TACACS Profiles and Command Sets are different, too

A simplified distinction is:

TACACS Profile: returns session attributes or privilege information for the device to interpret.
Command Sets: determine whether a requested command is permitted or denied.

Different products may use privilege levels, roles, or vendor-specific attributes. A profile named “Read-Only” is not proof of read-only behavior.

You need to understand how that device interprets the returned attributes and which authorization mechanisms it uses.

I plan to cover privilege levels, roles, and custom attributes separately.

Once the right Command Sets are selected, inspect their evaluation

Suppose the request arrived and the expected policy and Command Sets were selected. Now examine the command and arguments the device actually sent.

In the Cisco ISE 3.5 guide, the Command field uses wildcard matching, while Arguments uses regular expressions.

Do not assume the same expression means the same thing in both fields.

An ordinary Deny does not always win across multiple Command Sets

Imagine two sets are evaluated:

Set A returns an ordinary Deny for a command.
Set B returns Permit for the same request.

It is easy to assume that A must win. That is not how the documented multi-set evaluation works.

In brief:

A matching Deny Always takes precedence.
Without one, a Permit from any evaluated set allows the command.
Otherwise, it is denied, subject to the unmatched-command permission setting.

Within a set, matching and rule order also matter. Inspect the actual request and the complete combination of sets.

The answer is not to replace every Deny with Deny Always. First determine what matched, what permitted the request, and whether permission for unmatched commands affected the result.

GUI access may use a different authorization mechanism

Restricting commands in the CLI does not prove that the same account is read-only in the GUI.

Cisco's Catalyst 9800 configuration article, based on C9800-CL 17.9.2 and ISE 3.2.0, documents these WebUI restrictions:

Privilege levels 1–14 can access only the Monitor tab.
Privilege level 15 provides full access.
Using privilege level 15 with a restricted Command Set to enforce WebUI read-only access is unsupported; configuration changes may still be possible through the WebUI.

That is a very different situation from a typo in a Command Set.

Do not generalize the example to every WLC, release, or access method. Check support at the level of:

Product × OS version × access method (CLI, GUI, API, and so on).

My troubleshooting checklist

What you have established	What to check next
TACACS+ login succeeds	Authorization request for the target operation
No corresponding authorization request is visible	Device configuration, supported method, and log filters
An unexpected policy rule matched	Policy conditions and evaluation order
The expected policy rule matched	Selected Profile / Command Sets
The expected Command Sets were selected, but the operation was allowed	Command, Arguments, multiple-set evaluation, and unmatched-command handling
The CLI is restricted, but the GUI allows changes	GUI authorization mechanism and product documentation
ISE shows Deny, but the operation appears to have succeeded	Same operation and session? Was a change actually applied?

This turns “Command Sets don't work!” into something closer to:

The authorization request reaches ISE and matches the expected rule, but the result of evaluating multiple Command Sets differs from what we intended.

That is easier to investigate—and easier for TAC to respond to.

The same principle applies to read-only testing

If the goal is a read-only account, successful login is not the end of the test.

Check both sides:

Operations that should work	Operations that should be denied
Required show commands	Configuration changes
Troubleshooting checks	Reloads
Required log access	Saving configuration or changing users

Choose the actual test cases from your requirements and the product's supported controls. This table is illustrative, not a universal permissions list.

For tests involving configuration changes, saving, or reloading, assess the impact and recovery procedure in case the operation is unexpectedly allowed. Use a lab or another authorized test environment. Do not try such operations on production equipment on the assumption that “they should be denied.”

If people use both the CLI and GUI, test each access method.

The next topic in this series is why “the read-only user can log in” is not enough to finish an access-control test.

Follow the operation, not just the configuration screen

The lesson I took from working with Command Sets was simple:

“I created the configuration” and “that configuration controls this operation” are different claims.

Before changing Permit/Deny rules, work through these questions:

What operation was unexpectedly allowed?
Did its authorization request reach ISE?
Which policy rule matched?
Which Profile / Command Sets were selected?
What command and arguments did the device send?
How were multiple Command Sets evaluated?
Does that access method use the mechanism you are investigating?

Looking only at the Command Set configuration screen cannot answer all of them.

That is this week's field memo.

Original Japanese article: ISEでCommand Setsを作ったのに効かない？

This is an English adaptation of my Japanese article, prepared with AI translation assistance. Examples are generalized and contain no customer configurations or logs.

Official references

Cisco ISE 3.5 — Device Administration: Command Sets, matching, evaluation, and TACACS Profiles.
RFC 8907 — TACACS+: authentication, authorization, and accounting.
Cisco — Configure RADIUS & TACACS+ for GUI & CLI Authentication on Catalyst 9800 WLCs: the version-specific WebUI example.

Support and behavior depend on the product, OS version, and access method. Check the official documentation for the version you actually use.

snmpwalk Works. Is Your Monitoring Actually Ready?

NW Field Memo — Sun, 06 Sep 2026 09:17:12 +0000

Adapted from my original Japanese article, with AI-assisted translation and editing.

My manager: “Test this device.” (Doesn't really know the product or the technology.)

Me: “Sure.” (Also doesn't really know the product or the technology.)

If you've worked in infrastructure, that may sound familiar. I'm Goda, a network engineer sharing things I learned while figuring out the job.

SNMP comes up in a lot of network device testing. For a while, my idea of an SNMP test was simple:

Run snmpwalk.
Watch a pile of OIDs and values scroll past.
Mark SNMP as working.

A screen full of output is reassuring. It certainly looks like something is being monitored.

Then I was asked to write a test plan for a device. I added an item along the lines of “Confirm that information can be retrieved using SNMP” and sent it for review.

The feedback was:

Which OIDs will you use for CPU and memory?

The customer will probably ask. You should at least cover those.

That was when it clicked: a successful walk and successful retrieval of the metrics we need are two different things.

~~My other thought was, “Fine, you write the test plan, then.”~~

But the feedback was fair. Getting something back is not the same as getting what you need.

What did the successful walk actually prove?

snmpwalk is useful. Net-SNMP's tool uses GETNEXT requests to walk through a subtree starting from a specified OID. Net-SNMP manual

If it successfully returns values, you've established that you could read those values under those test conditions.

That matters. It does not, by itself, establish that your CPU and memory monitoring requirements are satisfied, or that every required OID is available.

I had been treating a successful command as a much broader result than it actually was.

Break “SNMP testing” into specific checks

Today, I'd separate at least these questions:

Question	What to check
Can I communicate over SNMP?	Whether the device responds to the intended request under the defined conditions
Can I monitor CPU?	The processor or resource being measured, the meaning of the value, and its averaging period
Can I monitor memory?	The resource being measured, its units, and any required conversion
Can the monitoring server collect the data?	Results from the actual monitoring server and monitoring application
Can I receive the required trap?	The notification produced by the relevant event and its documented trigger conditions

“SNMP polling: snmpwalk returns values” leaves a lot of room for interpretation.

It can be a perfectly reasonable connectivity check if you define its scope. If the goal includes CPU or memory monitoring, the acceptance criteria need to go further.

A number needs a definition

Suppose you retrieve:

What does it mean? Which resource does it describe? What are the units? Is it an instantaneous measurement or an average?

The following examples use HOST-RESOURCES-MIB to illustrate why those questions matter. Check what your actual device and software release support.

hrProcessorLoad reports an individual processor's non-idle time as a percentage, averaged over roughly the preceding minute. It is not simply an instantaneous, whole-device CPU percentage. The definition permits implementations to approximate that averaging period. RFC 2790

If it differs from the number on the device's GUI, first ask whether both displays measure the same thing over the same period.

For memory, hrStorageUsed is a count of allocation units. To express usage in bytes, multiply it by hrStorageAllocationUnits from the same table row:

Used bytes = allocation units in use × bytes per allocation unit

You also need to identify whether that row represents the memory area you intended to monitor. A readable OID is not enough; the interpretation needs to be right. RFC 2790

My laptop can read it. Can the monitoring server?

These are separate checks:

My laptop can retrieve the value.
The actual monitoring server can retrieve the value.
The monitoring application records it.
The application displays it as the metric we intended.

Don't sign off on more than your evidence supports.

A successful walk from a laptop does not establish that the monitoring system is working end to end.

That doesn't mean every device test has to cover the entire monitoring system. It means the test plan should state what this test covers, and what belongs to a later test or another team.

If end-to-end monitoring is in scope, the requirements may also call for checking threshold evaluation and alert delivery.

Network conditions matter, too. A laptop on the device's subnet and a monitoring server that reaches it through a router do not necessarily exercise the same path or access controls. Success from one does not prove access from the other.

Polling and traps need separate tests

Polling is the monitoring side asking, “What's the current value?”

A trap is the device reporting, “This event happened.”

Retrieval requests and SNMPv2 traps are distinct protocol operations. Reading an OID successfully does not prove that a trap exists for that condition, or that the device will send it in your test. RFC 3416

You need to check whether the product implements the notification and what triggers it.

Cisco ISE provides a useful documented example. The ISE 3.5 administration guide explains that manually stopping a process also stops Monit's monitoring of that process, so that operation does not produce the process-stop trap. It describes the trap as applying to an unexpected process stop that is not automatically recovered. Cisco ISE 3.5: Process-Monitoring SNMP Traps

So “I stopped the process and received no trap” is not enough to conclude that SNMP is broken.

This is an example from the ISE 3.5 documentation, not a rule for every product, release, or trap. Match your test to the documented behavior of the version you're testing.

How I'd write the test items now

My old test item was:

SNMP polling

Confirm that snmpwalk returns values.

I'd now separate it into more specific checks:

CPU monitoring

Retrieve the CPU-related metrics used by the monitoring design. Confirm that the measured resource, the meaning of the value, and the averaging period match that design.

Memory monitoring

Retrieve the memory-related metrics used by the monitoring design. Confirm that the resource, units, and any required conversion match that design.

Monitoring system collection

Collect the required metrics through the actual monitoring system and confirm that they are recorded and displayed as intended.

If the design specifies the OIDs, include them in the test plan. For table values, also record which instances to use and how to identify them.

That gives you an answer when someone later asks, “What exactly did we verify?”

A successful command is one piece of evidence

snmpwalk returns data. A ping succeeds. An SSH login works.

Each result is useful. Whether it is enough depends on the purpose of the test.

For monitoring, the question is whether we can monitor the things we're supposed to monitor. Anything outside the current test's scope should be explicitly handed over to the next test or the responsible team.

It sounds obvious now. I didn't really understand it until I had to write the test plan myself.

If your current SNMP test plan is basically “run a walk and see what happens,” I hope this gives you a few useful questions to add.

A related topic I plan to write about is whether knowing an OID tells you anything about the traps a device can send. That's another distinction I initially missed.