Foreword xxvi Introduction xxvii Part I Introduction to Troubleshooting Nexus Switches Chapter 1 Introduction to Nexus Operating System (NX-OS) 1 Nexus Platforms Overview 2 Nexus 2000 Series 2 Nexus 3000 Series 3 Nexus 5000 Series 4 Nexus 6000 Series 4 Nexus 7000 Series 5 Nexus 9000 Series 6 NX-OS Architecture 8 The Kernel 9 System Manager (sysmgr) 9 Messages and Transactional Services 11 Persistent Storage Services 13 Feature Manager 14 NX-OS Line Card Microcode 17 File Systems 19 Understanding NX-OS Software Releases and Packaging 25 Software Maintenance Upgrades 27 Licensing 28 NX-OS High-Availability Infrastructure 28 Supervisor Redundancy 29 ISSU 34 NX-OS Virtualization Features 35 Virtual Device Contexts 35 Virtual Routing and Forwarding 37 Virtual Port Channel 37 Management and Operations Capabilities 39 NX-OS Advanced CLI 39 Technical Support Files 44 Accounting Log 45 Feature Event-History 46 Debug Options: Log File and Filters 47 Configuration Checkpoint and Rollback 48 Consistency Checkers 49 Feature Scheduler, EEM, and Python 50 Bash Shell 51 Summary 51 References 51 Chapter 2 NX-OS Troubleshooting Tools 53 Packet Capture: Network Sniffer 53 Encapsulated Remote SPAN 57 SPAN on Latency and Drop 60 Nexus Platform Tools 63 Ethanalyzer 63 Packet Tracer 71 NetFlow 72 NetFlow Configuration 73 NetFlow Sampling 77 sFlow 78 Network Time Protocol 81 Embedded Event Manager 83 Logging 87 Debug Logfiles 90 Accounting Log 91 Event-History 92 Summary 93 References 93 Chapter 3 Troubleshooting Nexus Platform Issues 95 Troubleshooting Hardware Issues 95 Generic Online Diagnostic Tests 98 Nexus Device Health Checks 108 Nexus Fabric Extenders 124 Virtual Device Context 130 VDC Resource Template 131 Configuring VDC 133 VDC Initialization 134 Out-of-Band and In-Band Management 137 VDC Management 137 Troubleshooting NX-OS System Components 142 Message and Transaction Services 144 Netstack and Packet Manager 148 ARP and Adjacency Manager 160 EthPM and Port-Client 175 HWRL, CoPP, and System QoS 179 MTU Settings 192 Summary 195 References 196 Part II Troubleshooting Layer 2 Forwarding Chapter 4 Nexus Switching 197 Network Layer 2 Communication Overview 197 Virtual LANs 200 VLAN Creation 201 Access Ports 203 Trunk Ports 204 Private VLANS 207 Spanning Tree Protocol Fundamentals 218 IEEE 802.1D Spanning Tree Protocol 219 Rapid Spanning Tree Protocol 220 Multiple Spanning-Tree Protocol (MST) 236 Detecting and Remediating Forwarding Loops 241 MAC Address Notifications 242 BPDU Guard 243 BPDU Filter 244 Problems with Unidirectional Links 245 Summary 252 References 254 Chapter 5 Port-Channels, Virtual Port-Channels, and FabricPath 255 Port-Channels 255 Basic Port-Channel Configuration 259 Verifying Port-Channel Status 260 Verifying LACP Packets 262 Advanced LACP Configuration Options 265 LACP System Priority 268 Port-Channel Member Interface Consistency 271 Troubleshooting LACP Interface Establishment 272 Troubleshooting Traffic Load-Balancing 272 Virtual Port-Channel 274 vPC Fundamentals 275 vPC Configuration 278 vPC Verification 280 Advanced vPC Features 288 FabricPath 294 FabricPath Terminologies and Components 296 FabricPath Packet Flow 297 FabricPath Configuration 300 FabricPath Verification and Troubleshooting 303 FabricPath Devices 310 Emulated Switch and vPC+ 310 vPC+ Configuration 311 vPC+ Verification and Troubleshooting 314 Summary 320 References 320 Part III Troubleshooting Layer 3 Routing Chapter 6 Troubleshooting IP and IPv6 Services 321 IP SLA 321 ICMP Echo Probe 322 UDP Echo Probe 324 UDP Jitter Probe 325 TCP Connect Probe 328 Object Tracking 329 Object Tracking for the Interface 330 Object Tracking for Route State 330 Object Tracking for Track-List State 332 Using Track Objects with Static Routes 334 IPv4 Services 335 DHCP Relay 335 DHCP Snooping 341 Dynamic ARP Inspection 345 IP Source Guard 349 Unicast RPF 351 IPv6 Services 352 Neighbor Discovery 352 IPv6 Address Assignment 357 IPv6 First-Hop Security 362 First-Hop Redundancy Protocol 370 HSRP 370 VRRP 380 GLBP 385 Summary 391 Chapter 7 Troubleshooting Enhanced Interior Gateway Routing Protocol (EIGRP) 393 EIGRP Fundamentals 393 Topology Table 395 Path Metric Calculation 396 EIGRP Communication 399 Baseline EIGRP Configuration 399 Troubleshooting EIGRP Neighbor Adjacency 401 Verification of Active Interfaces 402 Passive Interface 403 Verification of EIGRP Packets 405 Connectivity Must Exist Using the Primary Subnet 409 EIGRP ASN Mismatch 412 Mismatch K Values 413 Problems with Hello and Hold Timers 414 EIGRP Authentication Issues 416 Troubleshooting Path Selection and Missing Routes 419 Load Balancing 421 Stub 421 Maximum-Hops 424 Distribute List 426 Offset Lists 427 Interface-Based Settings 430 Redistribution 430 Classic Metrics vs. Wide Metrics 433 Problems with Convergence 439 Active Query 441 Stuck in Active 443 Summary 446 References 447 Chapter 8 Troubleshooting Open Shortest Path First (OSPF) 449 OSPF Fundamentals 449 Inter-Router Communication 450 OSPF Hello Packets 450 Neighbor States 451 Designated Routers 452 Areas 453 Link State Advertisements 453 Troubleshooting OSPF Neighbor Adjacency 456 Baseline OSPF Configuration 456 OSPF Neighbor Verification 458 Confirmation of OSPF Interfaces 460 Passive Interface 461 Verification of OSPF Packets 463 Connectivity Must Exist Using the Primary Subnet 468 MTU Requirements 469 Unique Router-ID 471 Interface Area Numbers Must Match 471 OSPF Stub (Area Flags) Settings Must Match 473 DR Requirements 474 Timers 476 Authentication 478 Troubleshooting Missing Routes 482 Discontiguous Network 482 Duplicate Router ID 485 Filtering Routes 487 Redistribution 487 OSPF Forwarding Address 488 Troubleshooting OSPF Path Selection 494 Intra-Area Routes 494 Inter-Area Routes 495 External Route Selection 495 E1 and N1 External Routes 496 E2 and N2 External Routes 497 Problems with Intermixed RFC 1583 and RFC 2328 Devices 499 Interface Link Costs 500 Summary 504 References 505 Chapter 9 Troubleshooting Intermediate System-Intermediate System (IS-IS) 507 IS-IS Fundamentals 507 Areas 508 NET Addressing 509 Inter-Router Communication 511 IS Protocol Header 511 TLVs 512 IS PDU Addressing 512 IS-IS Hello (IIH) Packets 513 Link-State Packets 515 Designated Intermediate System 516 Path Selection 517 Troubleshooting IS-IS Neighbor Adjacency 518 Baseline IS-IS Configuration 518 IS-IS Neighbor Verification 520 Confirmation of IS-IS Interfaces 523 Passive Interface 526 Verification of IS-IS Packets 528 Connectivity Must Exist Using the Primary Subnet 535 MTU Requirements 537 Unique System-ID 539 Area Must Match Between L1 Adjacencies 539 Checking IS-IS Adjacency Capabilities 541 DIS Requirements 543 IIH Authentication 544 Troubleshooting Missing Routes 546 Duplicate System ID 546 Interface Link Costs 549 Mismatch of Metric Modes 553 L1 to L2 Route Propagations 556 Suboptimal Routing 562 Redistribution 566 Summary 567 References 568 Chapter 10 Troubleshooting Nexus Route-Maps 569 Conditional Matching 569 Access Control Lists 569 ACLs and ACL Manager Component 570 Prefix Matching and Prefix-Lists 577 Route-Maps 581 Conditional Matching 582 Optional Actions 586 Incomplete Configuration of Routing Policies 586 Diagnosing Route Policy Manger 586 Policy-Based Routing 591 Summary 594 References 595 Chapter 11 Troubleshooting BGP 597 BGP Fundamentals 597 Address Families 598 Path Attributes 599 Loop Prevention 599 BGP Sessions 600 BGP Identifier 601 BGP Messages 601 BGP Neighbor States 602 BGP Configuration and Verification 605 Troubleshooting BGP Peering Issues 609 Troubleshooting BGP Peering Down Issues 609 Demystifying BGP Notifications 619 Troubleshooting IPv6 Peers 621 BGP Peer Flapping Issues 622 BGP Route Processing and Route Propagation 630 BGP Route Advertisement 631 BGP Best Path Calculation 636 BGP Multipath 640 BGP Update Generation Process 643 BGP Convergence 646 Scaling BGP 649 Tuning BGP Memory 650 Soft Reconfiguration Inbound Versus Route Refres.