DEV Community

Nick Schmidt
Nick Schmidt

Posted on • Originally published at blog.engyak.net on

5 2

Troubleshooting with VMware NSX ALB/Avi Vantage

NSX Advanced Load Balancer - Logging and Troubleshooting Cheat Sheet

Get into the OS Shell (all elements)



sudo su


Enter fullscreen mode Exit fullscreen mode

Controller Log Locations

Note: Everything in /var/lib/avi/logs is managed by Elasticsearch. I wouldn't mess with it.

Events published to the GUI: /var/lib/avi/logs/ALL-EVENTS/

The primary log directory for Avi Vantage Controllers is /opt/avi/log. As this feeds into Elasticsearch, they have file outputs for every severity level. An easy way to get data on a specific object would be to build a grep statement like this:



grep {{ regex }} /opt/avi/log/{{ target }}


Enter fullscreen mode Exit fullscreen mode
  • alert_notifications_*: Summarized problems log. Events are in a json format!

Troubleshooting Deployment Failures

  • avi-nsx.*: Presumably for NSX-T integration. further investigation required
  • cloudconnectorgo.*: Avi's cloud connector is pretty important given their architecture. This is where you can troubleshoot any issues getting a cloud turned up, or any initial provisioning issues.
  • vCenter*: vCenter write mode activity logs. Look here for SE deployment failures in a traditional vSphere cloud.

Service Engines

Troubleshooting

Checking the Routing Table

NSX ALB / Avi uses FRRouting (7.0 as of release 20.1) over network namespaces to achieve management/data plane separation and VRF-Lite. To access the data plane, you will need to change namespaces! Unlike NSX-T, this doesn't happen over docker namespaces. This means that the follow commands work in both as root:

  • Show all VRF+Namespaces ip netns show
  • Send a one-shot command to the namespace: ip netns exec {{ namespace }} {{ command }} Example: ip netns exec 'ip route show'
  • Start a shell in the desired namespace: ip netns exec {{ namespace }} {{ shell }} Example: ip netns exec avi_ns1 bash

After in the bash shell, all normal commands apply as if there was no namespace/VRF.

For more information on Linux Network Namespaces, here's a pretty good guide: https://www.opencloudblog.com/?p=42

Logging

All SE logging is contained in /var/lib/avi/log. Here are the significant log directories there:

  • IMPORTANT! bgp: This is where all the routing protocol namespace logging from FRRouting lands.
  • traffic: This one's pretty touch to parse and it's better to use Avi's Elasticsearch instead.

Conclusion

Avi Vantage has a pretty solid logging schema, but is very much a growing product. These logs will eventually be exposed more fully to the GUI/API, but for now it's handy to grep away. I'll be updating this list as I find more.

Image of Timescale

Timescale – the developer's data platform for modern apps, built on PostgreSQL

Timescale Cloud is PostgreSQL optimized for speed, scale, and performance. Over 3 million IoT, AI, crypto, and dev tool apps are powered by Timescale. Try it free today! No credit card required.

Try free

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more