OVS Database and Flow Analysis

This skill provides detailed analysis of Open vSwitch (OVS) data collected via sosreport. It operates in four modes:

Default mode: Full analysis - conf.db + all text files (requires ovsdb-tool, falls back to text files if not available)
Database mode (--db): Database only - analyze conf.db (requires ovsdb-tool)
Text files mode (--flows-only): Text files only - no ovsdb-tool needed
Query mode (--query): Run custom OVSDB JSON queries (requires ovsdb-tool)

When to Use This Skill

Use this skill when:

Troubleshooting packet drops - Find drop flows being hit
Analyzing flow performance - View top flows by packet count
Analyzing OVS configuration from a sosreport
Troubleshooting bridge, port, or interface issues
Investigating DPDK vs kernelspace datapath
Reviewing tunnel configurations (VXLAN, Geneve, GRE)
Checking port statistics (drops, errors, traffic)
Mapping Kubernetes pods to OVS interfaces

Prerequisites

Default mode (full analysis):

ovsdb-tool must be installed (from openvswitch package)
Falls back to --flows-only if ovsdb-tool not found
Sosreport with sos_commands/openvswitch/ directory and conf.db

Database mode (--db):

ovsdb-tool must be installed (from openvswitch package)
- Check: which ovsdb-tool
- Fedora/RHEL: sudo dnf install openvswitch
- Ubuntu/Debian: sudo apt install openvswitch-common
Sosreport with conf.db file

Text files mode (--flows-only):

No special tools needed - works out of the box!
Sosreport with sos_commands/openvswitch/ directory

Query mode (--query):

ovsdb-tool must be installed
Incompatible with --flows-only

File Locations in Sosreport

The sosreport collects:

sosreport-hostname-date/
├── etc/openvswitch/conf.db              (OVS database - for --db mode)
├── var/lib/openvswitch/conf.db          (alternate location)
└── sos_commands/openvswitch/            (default mode uses these)
    ├── ovs-vsctl_-t_5_show              (topology)
    ├── ovs-vsctl_-t_5_list_bridge       (bridge details)
    ├── ovs-vsctl_-t_5_list_interface    (interface details)
    ├── ovs-vsctl_-t_5_list_Open_vSwitch (system info)
    ├── ovs-ofctl_dump-flows_<bridge>    (OpenFlow entries)
    ├── ovs-ofctl_dump-ports_<bridge>    (Port statistics)
    ├── ovs-appctl_coverage.show         (internal stats)
    ├── ovs-appctl_upcall.show           (datapath health)
    ├── ovs-appctl_tnl.ports.show_-v     (tunnel ports)
    └── ...

Analysis Script

The analysis script is located at:

<plugin-root>/skills/ovs-db-analysis/scripts/analyze_ovs_db.py

Implementation Steps

Step 1: Locate the Analysis Script

SCRIPT_PATH=$(find ~ -name "analyze_ovs_db.py" -path "*/sosreport/skills/ovs-db-analysis/scripts/*" 2>/dev/null | head -1)

if [ -z "$SCRIPT_PATH" ]; then
    echo "ERROR: analyze_ovs_db.py script not found."
    exit 1
fi

Step 2: Run the Analysis

# Default: Full analysis - conf.db + text files (requires ovsdb-tool)
python3 "$SCRIPT_PATH" /path/to/sosreport-hostname-date/

# Analyze from archive
python3 "$SCRIPT_PATH" /path/to/sosreport-hostname-date.tar.xz

# Database only mode (requires ovsdb-tool)
python3 "$SCRIPT_PATH" /path/to/sosreport/ --db

# Text files only mode (no ovsdb-tool needed)
python3 "$SCRIPT_PATH" /path/to/sosreport/ --flows-only

Step 3: Custom Queries (Optional, requires --db)

For specific investigations, use raw OVSDB queries:

# Query all bridges
python3 "$SCRIPT_PATH" /path/to/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Bridge", "where":[], "columns":["name","datapath_type","fail_mode"]}]'

# Query VXLAN interfaces
python3 "$SCRIPT_PATH" /path/to/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Interface", "where":[["type","==","vxlan"]], "columns":["name","options"]}]'

# Query interfaces with errors
python3 "$SCRIPT_PATH" /path/to/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Interface", "where":[], "columns":["name","error"]}]'

Analysis Output

The default mode analyzes:

| Analysis | Source File | Description | |----------|-------------|-------------| | System Info | ovs-vsctl_-t_5_list_Open_vSwitch | OVS/DPDK version, system type, external IDs | | Topology | ovs-vsctl_-t_5_show | Bridge overview with ports grouped by type | | Bridge Details | ovs-vsctl_-t_5_list_bridge | Datapath type, fail mode, CT zones | | Interfaces | ovs-vsctl_-t_5_list_interface | By type, pod interfaces with mapping | | OpenFlow | ovs-ofctl_dump-flows_* | Flow counts, drops, top flows | | Port Stats | ovs-ofctl_dump-ports_* | RX/TX drops and errors | | Tunnels | ovs-appctl_tnl.ports.show_-v | Configured tunnel ports | | Datapath Health | ovs-appctl_upcall.show | Flow table usage vs limit | | OVS Stats | ovs-appctl_coverage.show | Internal counters (netlink, OVSDB, etc.) |

OpenFlow Analysis Features

| Feature | Description | |---------|-------------| | Flow Count | Total flows per bridge (total, drop, with hits) | | Drop Detection | Flows with actions=drop that have packet hits | | Top Flows | Most active flows sorted by n_packets | | Table Distribution | Flow counts per OpenFlow table | | Port Drops | RX/TX drop counters per port | | Port Errors | RX/TX error counters per port | | Datapath Health | Flow table usage vs limit (from upcall.show) | | OVS Stats | Internal statistics (netlink, OpenFlow, OVSDB transactions) |

OVN Internal Tables (Ignore Drops in These)

When analyzing drop flows in OVN-managed bridges (br-int), ignore drops in table 44 and tables 64-87 as these are internal OVN mechanics, not security policy or real packet drops.

⚠️ Note: OVN and OVS are complex systems and table mappings can change between releases. Always verify drop analysis against the specific OVN version in use. When in doubt, check the OVN source code for your release.

| Table | Name | Purpose | |-------|------|---------| | 44 | CHK_LB_OUTPUT | Loopback prevention - drops packets that would loop back (high volume, normal) | | 64 | SAVE_INPORT | Save ingress port for later | | 65 | LOG_TO_PHY | Logical to physical mapping | | 66 | MAC_BINDING | MAC binding lookups | | 67 | MAC_LOOKUP | MAC address table lookups | | 68-69 | CHK_LB_HAIRPIN | Load balancer hairpin checks | | 70 | CT_SNAT_HAIRPIN | Conntrack SNAT hairpin | | 71-72 | GET/LOOKUP_FDB | FDB (forwarding DB) lookups | | 73-74 | CHK_IN_PORT_SEC | Ingress port security | | 75 | CHK_OUT_PORT_SEC | Egress port security | | 76-77 | ECMP_NH | ECMP next-hop handling | | 78 | CHK_LB_AFFINITY | Load balancer affinity | | 79 | MAC_CACHE_USE | MAC cache miss (high volume, normal) | | 80 | CT_ZONE_LOOKUP | Conntrack zone lookup | | 81-83 | CT_ORIG_*_LOAD | Conntrack original tuple loading | | 84 | FLOOD_REMOTE_CHASSIS | Remote chassis flooding | | 85 | CT_STATE_SAVE | Conntrack state save | | 86 | CT_ORIG_PROTO_LOAD | Conntrack protocol loading | | 87 | GET_REMOTE_FDB | Remote FDB lookup |

Relevant drop tables for troubleshooting:

Table 9: Ingress ACLs - actual policy drops (look for reg0=0x8000)
Table 0: Initial classification drops
Tables < 64: Generally meaningful drops

Common Analysis Scenarios

1. Troubleshooting Packet Drops

# Quick analysis (default mode, no ovsdb-tool)
python3 "$SCRIPT_PATH" /path/to/sosreport/

Look for:

ACTIVE DROP FLOWS section showing flows dropping packets
PORT STATISTICS showing ports with drops/errors

2. Checking DPDK Configuration

# Default mode shows DPDK info from text files
python3 "$SCRIPT_PATH" /path/to/sosreport/

# Or use database mode for custom queries
python3 "$SCRIPT_PATH" /path/to/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Open_vSwitch", "where":[], "columns":["dpdk_initialized","other_config"]}]'

3. Investigating Pod Connectivity

The interface analysis shows pod-to-OVS mapping:

Interfaces with external_ids containing iface-id
Interface names ending with _h (veth host side)

4. Datapath Health Check

The upcall stats show:

DATAPATH FLOW TABLE HEALTH
  Current flows: 155 / 200,000 (0.1% used)
  Average: 156, Max seen: 215
  ✓ Flow table healthy

If usage > 90%, flows are being evicted too aggressively.

Output Example

================================================================================
OVS ANALYSIS - sosreport-hostname-2024-01-15
================================================================================
Mode: Text file analysis (no ovsdb-tool required)

================================================================================
OVS SYSTEM INFORMATION
================================================================================

  Field                     Value
  ------------------------- --------------------------------------------------
  OVS Version               "3.3.4-62.el9fdp"
  DB Version                "8.5.0"
  System Type               rhcos
  System Version            "4.16"
  DPDK Initialized          false
  Datapath Types            [netdev, system]

  External IDs:
    hostname: master2.example.com
    ovn-encap-ip: 10.32.110.5
    ovn-encap-type: geneve

================================================================================
OVS TOPOLOGY
================================================================================

  System UUID: 7e9a3f70-86fa-4578-a849-4fd807a64a10
  Total Bridges: 2

  Bridge: br-ex
    ports: 3
      internal: br-ex
      patch: patch-br-ex-to-br-int
      system: nm-bond

  Bridge: br-int
    fail_mode: secure
    datapath_type: system
    ports: 12
      geneve: 9 ports
      internal: ovn-k8s-mp0, br-int
      patch: patch-br-int-to-br-ex

================================================================================
OPENFLOW ANALYSIS
================================================================================

  Bridge: br-int
  ----------------------------------------------------------------------
  Total flows: 2,017
  Flows with hits: 318
  Drop flows: 150 (9 actively dropping)

  ⚠️  ACTIVE DROP FLOWS (9):
    table=40, priority=0, packets=8,105
    table=79, priority=100, packets=1,356
      match: ip,reg14=0x2,metadata=0x5,dl_src=00:62:0b:ea:b5:e0

--------------------------------------------------------------------------------
DATAPATH FLOW TABLE HEALTH
--------------------------------------------------------------------------------

  Current flows: 155 / 200,000 (0.1% used)
  Average: 156, Max seen: 215
  ✓ Flow table healthy

Error Handling

| Error | Solution | |-------|----------| | ovsdb-tool not found | Install openvswitch package OR use default mode (no --db) | | conf.db not found | Use default mode (analyzes text files instead) | | sos_commands/openvswitch not found | Ensure sosreport has OVS data collected |

OVS Database and Flow Analysis

OVS Database and Flow Analysis

When to Use This Skill

Prerequisites

File Locations in Sosreport

Analysis Script

Implementation Steps

Step 1: Locate the Analysis Script

Step 2: Run the Analysis

Step 3: Custom Queries (Optional, requires --db)

Analysis Output

OpenFlow Analysis Features

OVN Internal Tables (Ignore Drops in These)

Common Analysis Scenarios

1. Troubleshooting Packet Drops

2. Checking DPDK Configuration

3. Investigating Pod Connectivity

4. Datapath Health Check

Output Example

Error Handling

See Also