DEV Community

Manoel Martins
Manoel Martins

Posted on

What my automation is never allowed to do

#networking #automation #python #homelab

Tested on NetBox 4.5, FortiOS 7.4, RouterOS 7.23 (long-term) and Proxmox VE 8.3

I wanted NetBox to reflect what my lab actually runs, not what I remembered to type into it. So I wrote a script that reads a FortiGate, a MikroTik and a Proxmox host every night and reconciles NetBox with what they report.

Writing the collectors took an afternoon. The hard part was a different question: this thing runs unattended, every day, with write access to my source of truth. What must it never do?

I ended up with four rules. Each one I either tested on purpose or learned by watching it fail.

The setup 

FortiGate (REST API)   ─┐
MikroTik (RouterOS API)─┼─(read-only)─> netbox-sync ─(write)─> NetBox
Proxmox VE (API token) ─┘                    │
                                             └─ local network sweep (ping + ARP)
Enter fullscreen mode Exit fullscreen mode
Source What it reports
FortiGate serial, interfaces, VLANs, MACs, IPs, DHCP pools, ARP
MikroTik serial, interfaces, VLANs, MACs, IPs, DHCP pools, ARP
Proxmox VE VMs and containers, vCPU, RAM, disks, interfaces, host capacity

It runs from a systemd timer once a day, and on demand from a double-click shortcut. Every run that changes something writes a Journal Entry on the NetBox device.

Rule 1 — Least privilege: the devices are only read

The script never needs to change the network, so it never gets the ability to.

Platform Account
FortiGate API admin with a read-only accprofile, trusthost pinned to the sync host
MikroTik group with read,api and without write, policy or sensitive; user bound to the sync host's IP
Proxmox VE API token with the built-in PVEAuditor role

The only credential that can write anything is the NetBox token — also restricted to the sync host's IP.

I tested it instead of trusting it. A POST to start a container on Proxmox with the sync token:

403 Permission check failed (/vms/101, VM.PowerMgmt)
Enter fullscreen mode Exit fullscreen mode

On MikroTik, leaving out sensitive has a nice side effect: the account can read the whole configuration but not the Wi-Fi keys or VPN secrets stored in it.

If the script is ever wrong — or its config file leaks — the worst case is a wrong inventory. Not a wrong firewall.

Rule 2 — Retire, never delete

The naive version of a sync is: if it's in NetBox but not on the device, delete it.

That rule is fine until an API call times out. Then one bad night wipes good data, and the next morning your source of truth says the VLAN never existed.

"Missing from the source" is not the same as "gone". So a normal run never deletes. It retires:

Gone from the source Normal run does If it comes back
DHCP pool IP Range → deprecated, no longer counted as used reactivated
Proxmox VM VM → decommissioning status restored
Device interface alert in the log —

Deleting is a separate, explicit step: --prune. It never runs from the timer, it is not available through the on-demand shortcut, and it only deletes objects the script itself created — tagged [netbox-sync]. Anything I entered by hand is out of its reach, even when it is retired.

I tested this without touching a single device, by creating fake objects in NetBox:

Fake object Tag Normal run --prune
VM + interface + IP yes retired deleted, with its IP
VM no (manual) retired kept
DHCP IP Range yes retired, range free again deleted

A second normal run changed nothing. It only reminded me what was waiting for a decision.

Rule 3 — A dry-run must show everything

--dry-run is the command I run before I trust anything. My first version printed this when I created a test VLAN on each device:

[change] interface created: vlan999 (vlan)
[change] interface created: vlan999 (vlan)
Enter fullscreen mode Exit fullscreen mode

Both VLANs had an IP address and a parent interface. Apply would have created those too. The preview simply didn't say so.

A preview that shows less than the real run is worse than no preview: it gives you confidence you didn't earn. The fixed version prints every change apply will make:

fw-01
  [change] interface created: vlan999 (vlan VLAN ID 999 on internal3)
  [change] vlan999: IP created 192.0.2.1/24
rtr-01
  [change] interface created: vlan999 (vlan VLAN ID 1 on ether3)
  [change] vlan999: IP created 192.0.2.2/24
Enter fullscreen mode Exit fullscreen mode

Look at the second block. The MikroTik interface was named vlan999 — but it was tagging traffic as VLAN 1. RouterOS uses vlan-id=1 when you don't specify one. The name looked right, so nobody would have noticed. The old dry-run would have hidden it too.

Fixing the preview found a real misconfiguration on its first run.

Rule 4 — "Free" must mean free

The reason to have an IPAM is to answer one question safely: which address can I use?

NetBox was happy to suggest addresses inside my DHCP pools — addresses the DHCP server could hand to a phone at any moment. So active DHCP pools are now recorded as one IP Range each, with two flags set:

  • mark_populated — the range disappears from "available IPs"
  • mark_utilized — the range counts toward prefix utilization

A pool counts as active only if both the DHCP server and its interface are up. My firewall had five DHCP servers configured on SSIDs that were switched off; those are recorded as deprecated, not as addresses in use.

Individual leases are never recorded. They are volatile, and a lab VM that lives for a week on DHCP should leave no trace in the inventory.

The second half of the rule: an inventory only knows what it sees. A host with a static IP that nobody documented is invisible to every API. So each run also sweeps the directly connected networks — a ping to every address, then a look at the sync host's own ARP table — and alerts on anything answering on the wire that NetBox doesn't know. About 9 seconds for a /24. Addresses inside DHCP pools never alert.

After both changes, asking NetBox's API for available IPs gave the right answer — including an uncomfortable one: on one of my networks, the gateway plus the DHCP pool cover the whole /24. There is no free address for a static host there. A design finding I would otherwise have made while configuring a device.

What the first runs found

Things my manual inventory had missed, in the first few days:

  • A gateway IP that was never documented. The first run created it.
  • An orphan DHCP server on the MikroTik, still pointing to a bridge I had deleted weeks earlier. RouterOS shows the missing interface as *C; the script flags that as an orphan.
  • VM disks under-reported. Two VMs have a second disk (one of them a virtual firewall with separate boot and log disks). My manual entry had counted only the first.
  • A VM wearing the host's MAC. Found by the network sweep, outside Proxmox: a VM bridged over Wi-Fi on VMware Workstation answers ARP with the host's Wi-Fi MAC — 802.11 doesn't let one client transmit with several MACs, so the hypervisor rewrites it. Any MAC-based troubleshooting of that VM would point at the laptop.
  • RAM overcommit on the hypervisor, reported on every run: allocated vs. physical, with an alert above 100%.

Things I learned the hard way

  • Line endings. I edited the script on Windows and saved it with CRLF. Linux read the #! line with an invisible \r and answered cannot execute: required file not found. py_compile passed — Python doesn't care. The systemd timer would have failed silently that night. The repository now has a .gitattributes that forces LF.
  • API endpoints move between versions. On FortiOS 7.4 the ARP table is /api/v2/monitor/network/arp; /monitor/system/arp returns 404. My first attempt took the whole FortiGate collection down with it — now auxiliary data like ARP is isolated, and a failure there never stops the rest.
  • Test the silence. When the ARP check first returned zero alerts, it wasn't because the network was clean. Device ARP tables only list hosts that talked to the device recently. Zero alerts meant zero coverage. That is why the local sweep exists.

Known limits

  • IPs of Proxmox VMs are not collected — that needs the qemu-guest-agent inside each VM. Container IPs are.
  • One FortiGate VDOM per source entry.
  • Devices without an API — an unmanaged switch, the ISP router, consumer Wi-Fi — stay manual.

Built with Claude as a pair-programmer — every change was validated against the real devices before going live.

The code is on GitHub under the MIT license, with the read-only account setup for each platform, the design notes and the test scenarios: github.com/manoelmartins1980/netbox-sync

What's the one thing your automation is never allowed to do?

Top comments (0)