Troubleshoot topology by designing the as-is map, breaking L2 loops with Rapid PVST/RSTP, load-balancing uplinks with LACP/ECMP, fixing switch misconfigurations (VLANs/routing), and offering backup links on critical segments. Record the new design so that future changes are predictable.
Symptoms
- Random slowdowns or timeouts on segments that ping fine locally but fail across VLANs
- Broadcast storms: switch LEDs blinking in unison, CPU on switches near 100%
- Single link failure takes down large portions of the LAN instead of failing over
- Duplicate MAC address warnings or MAC flapping messages in switch logs
- Traceroute shows asymmetric or unexpectedly long paths between two internal hosts
Common Causes
Open Layer 2 loops:
When two switches are connected with parallel cables and without using STP or LACP. Frames go around and around forever, until they saturate each uplink and bring down the network.
Abnormal topology (missing or mis-configured Spanning Tree):
STP not enabled on access switches, or root bridge chosen incorrectly can cause suboptimal paths, and twitchy convergence when links go down or come up.
Flat network, no segmentation:
Broadcast domain of hundreds of hosts increases ARP traffic and broadcast. Small fault has a large effect over the entire site.
Single points of failure on critical uplinks:
Core-to-distribution links that use a single cable, a single transceiver, or a single power supply kill entire VLANs or entire buildings in the event of hardware failure.
Unbalanced traffic and hot-spot uplinks:
Same access switch, same rack, servers and sensitive clients often share the same access switch uplinks, which results in one uplink getting saturated and the other lying fallow, often because ECMP or LACP hashing was never optimized.
Step-by-Step Fix
- Map your existing network - not the one on the old drawing:
Use show lldp neighbors and show cdp neighbors detail on a managed switch, along with an ARP sweep or nmap -sn on each subnet, and import the information into a network-mapping platform such as NetBox, LibreNMS, SolarWinds NTM, etc. Plan a diagram that reflects reality for the port. - Find and break Layer 2 loops before touching anything else
Check show spanning-tree for TCN storms and unexpected root bridges. Inspect switch CPU with show processes cpu sorted. If two access switches are cross-connected without a port-channel, shut one link and rebuild it as an LACP bundle. Confirm Rapid PVST or MST is enabled on every switch, not only the core. - Pick the definitive root bridge, secure it:
Select the central switch that will host the spanning-tree root and set its priority (spanning-tree VLAN 1-4094 priority 4096). Set the second core to 8192. Turn on BPDU Guard on access ports and Root Guard on downlinks so a rogue switch does not take over the topology. - VLANs and routed uplinks were able to isolate the campus:
Split large broadcast domains on the campus into functions: users, voice, servers, management, guest. Terminate each VLAN on a Layer 3 distribution switch, instead of trunking it across the campus. Smaller broadcast domains decrease the blast radius of future misconfigurations and greatly reduce troubleshooting time. - Add redundancy on links that cannot fail quietly
For any core, distribution, or server uplink, provision a second physical path over a different line card or switch. Bond parallel links with LACP for active/active use, or rely on STP for active/standby. Test failover by pulling one cable during a maintenance window and confirming sub-second recovery. - Rebalance Traffic over Multiple Paths
Validate Links and look for latents working hard and the-others-idle. Show interfaces summary will give you interface counters. Optimize LACP load-balancing hash (src-dst-ip, src-dst-port) to balance across member links. Layer 3 enable ECMP and check show ip route to ensure multiple next-hops are installed. - Test end-to-end paths and continue watching:
Use traceroute to test connectivity between appropriate hosts in each VLAN and ensure the path is as expected. Use SNMP or streaming telemetry in an application such as LibreNMS, Zabbix, or PRTG. Set monitoring thresholds for link utilization greater than 70%, STP topology changes, and duplex mismatch for early detection of drift. - Document the corrected topology and change process
Update the diagram, VLAN table, IP plan, and port descriptions on every switch (interface description, snmp location). Store the config in Git or RANCID. A topology fix that is not documented will regress the next time someone patches a cable at 2 a.m.
Common topology symptoms mapped to likely cause and first fix
| Symptom | Likely cause | First action |
|---|---|---|
| Switch CPU pinned, LEDs blinking together | Layer 2 loop, broadcast storm | Shut suspected redundant port, verify STP is active |
| One uplink saturated, parallel link idle | LACP hash or ECMP not tuned | Change port-channel load-balance method |
| Single cable pull kills a VLAN | No redundant path on that segment | Add second uplink via different switch or line card |
| MAC flapping log entries | Loop or duplicated MAC on two ports | Locate offending ports with show mac address-table |
| Traceroute takes unexpected long path | Suboptimal STP root or missing route | Set root bridge priority, review routing table |
| Broadcast traffic overwhelms clients | Flat, unsegmented network | Introduce VLANs and route between them |
Prevention
- Enforce a change-control rule: no new switch uplink goes live without LACP or STP verification
- Review capacity and link utilization quarterly, not only after an outage
- Keep VLAN, IP, and cable-label documentation in one system, updated on every change
- Standardize on one STP mode (RSTP or MST) across the whole estate to avoid interop bugs
FAQ
How to define the difference between layer 2 and 3 topology problem?
L2 problems typically appear as broadcast storms, MAC flaps, or STP topology-change indications, and they span a single VLAN. L3 problems appear as routing loops, asymmetric traceroutes, or inaccessibility, which is host-GW-dependent. First, check show spanning-tree and switch CPU; if both are OK, proceed to routing tables and OSPF or BGP neighbor states.
Will the total mesh, all-sites connected, be better than the hub and spoke/star design?
No. Full mesh does not scale well since the link count is quadratic in the number of nodes, and it makes STP and routing more difficult.
How to repair topology without service interruption?
Partially. Live additions: unused uplink, BPDU Guard, LACP delay/configure hashing. Live replacements: move root bridge; renumber VLANs; re-terminate trunk.