Cisco Nexus 6000 and 5600 with Fabric Extender 2000 Switch

Cisco Nexus 6000 and 5600 with Fabric Extender 2000 Switch Architecture BRKARC-3452

Nicolas Delecroix Technical Marketing Engineer Data Center Group – Nexus Business Unit [email protected]

Nexus 5000 Evolution

Nexus 5020 56-Port, 2RU Customers

Nexus 5010 28-Port, 1RU BRKARC-3452

Nexus 5548 48-Port, 1RU

Nexus 5596T 10Gbase-T, 2RU

Nexus 5596 96-Port, 2RU © 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

3

Introducing Cisco Nexus 5600/6000

Nexus 6004 Nexus 5020 56-Port, 2RU Customers





Cisco Public

4

96 port 40G, 4RU

Introducing Cisco Nexus 5600/6000 Nexus 5672 48 port 10G, 6 port 40G, 1RU

Nexus 56128 Nexus 6004 96 port 10G, 8 port 40G, 2RU Nexus 5020 56-Port, 2RU Customers





Cisco Public

5

96 port 40G, 4RU

Cisco Nexus 6000 and 5600 with Fabric Extender 2000 Switch Architecture BRKARC-3452

Nicolas Delecroix Technical Marketing Engineer Data Center Group – Nexus Business Unit [email protected]

Nexus 5600/6000 Architecture - Agenda Overview Fabric Extender Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications BRKARC-3452

© 2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

7

Nexus 5672 Chassis Port-Side View • All ports provide Ethernet and FCoE • 16 Unified Ports (orange) also provide 2/4/8G FC • L2 and L3 VXLAN

48 fixed 1/10G SFP+ interfaces BRKARC-3452


Cisco Public

8

6 fixed 40G QSFP interfaces

Cisco Nexus 5672 Chassis Power Supply and Fans Power Supply 1+1 Redundancy

Fan Module 2+1 Redundancy

BRKARC-3452


Cisco Public

Console Mgmt0 USB

9

Nexus 56128 Chassis Port-Side View • 48 Fixed 10G SFP+ Ports, 4 fixed 40G QSFP+ ports

• 2 Expansion Slots • Expansion module provides 24 Unified ports 10G Ethernet/FCoE or 2/4/8G FC, and 2 ports 40G Ethernet/FCoE

BRKARC-3452


Cisco Public

10

Cisco Nexus 56128 Chassis Power Supply and Fans Power Supply 2+2 Redundancy

Fan Module 3+1 Redundancy

Console Mgmt0 USB

BRKARC-3452


Cisco Public

11

Cisco Nexus 6004 Chassis 12 QSFP+ ports Expansion Module N6K-6004-M12Q

Port-Side View

4RU

BRKARC-3452


Cisco Public

12

Cisco Nexus 6004 Chassis Power Supply and Fans • Six power supply slots; a minimum of three is required. Supports both 3 + 1 and 3 + 3 redundancy. • Each power supply is rated 1100W, 90—240 VAC. • The chassis has four fan trays. A minimum of three is required.

Power Supply 3 + 3 Grid Redundancy or 3 + 1 Redundancy

Console Mgmt0 USB

Fan Module 3+1 Redundancy BRKARC-3452


Cisco Public

13

Unified Port Line-Card Expansion Module - N6004X-M20UP For Nexus 6004 • 20 * 10G SFP+ unified ports expansion module • 2/4/8G FC port • 1G/10G and FCoE • Max number of native FC ports per chassis is 48 (as of NXOS 7.0(2)N1(1) release) • SFP+ port allows support for a larger variety of optical transceivers

• Same transceivers supported as on 5600 10G ports – including 10G long distance, 1G copper and optical BRKARC-3452


Cisco Public

14

Cisco Nexus 5600/6000 Airflow • The units have a “port side” and a “fan side”. No front or back • The units support both port side exhaust and port side intake – Port side exhaust (used for in server rack airflow alignment) – Port side intake (used for network rack airflow alignment)

• Different power supply and fan modules are required for different airflow directions

BRKARC-3452


Cisco Public

15

N5600/6000 and N5500 Comparison Feature

Nexus 5600/6000

Nexus 5500

L2 Throughput

Line rate

Line rate

L3 Throughput

Line rate

160Gbps

Latency

~1us

1.8us

Native 40GE Support

Yes

4xQSFP module – 10G flows

1GE Support

Yes

Yes

Unified Ports FEX Scaling

Yes 48 (6004 L2), 24 (6004 L3, 5600)

Yes 24 with L2, 16 with L3

MAC Table IPv4 Unicast Routes

256K (shared with ARP/ND) 32K

32K 16K

IPv4 Multicast Routes

32K

8K (shared with ARP)

IPv4 ARP/Host Route Entries

128K

16K (shared with IPv6 ND and mroute)

BRKARC-3452


Cisco Public

16

N5600/6000 and N5500 Comparison (cont.) Feature

Nexus 5600/6000

Nexus 5500

IPv6 Unicast Routes

8K (shared with IPv4)

8K (shared with IPv4)

IPv6 ND Entry/Host Route Entries

85K (shared with MAC and ARP)

8K (shared with mroute and ARP)

IPv6 Multicast Routes

32K

8K (shared with ARP, ND. Limit is with /64 routes)

VLAN

4K

4K

VRF

4K

1K

Segment ID

Yes

No

ACL

4K

4K

L4ops for ACL

24 total (16 for UDP/TCP, 8 for TCP flag)

8 for UDP and 8 for TCP

Packet Buffer

640KB per 10G port

vPC, FabricPath

25MB per 3xQSFP (or 12xSFP) ports 2.1MB per 10G port Yes

Dynamic Fabric Automation

Yes

No

VXLAN

Yes*

No

*Software roadmap feature BRKARC-3452


Cisco Public

17

Yes



Cisco Public

18

Nexus 2000 100M/1G Fabric Extender Portfolio

Nexus 2224TP

Nexus 2248TP

Nexus 2248TP-E

2 x 10G SFP+

4 x 10G SFP+

4 x 10G SFP+

Host Facing Ports

24 x 100/1000BASE-T RJ45

48 x 100/1000BASE-T RJ45

48 x 100/1000BASE-T RJ45

Oversubscription

1 : 1.2

1 : 1.2

1 : 1.2

FCoE

N/A

N/A

N/A

Buffer

3.8MB

7.6MB

32MB

No

No

Yes

Uplink Ports

Shared Buffer

BRKARC-3452


Cisco Public

19

Nexus 2000 1G/10G Fabric Extender Portfolio

Nexus 2232PP

Nexus 2232TM

Nexus 2232TM-E

Nexus 2248PQ

8 x 10G SFP+

8 x 10G SFP+

8 x 10G SFP+

4 x 40G QSFP+

Host Facing Ports

32 x SFP/SFP+ (1/10G)

32 x 1/10GBASE-T RJ45

32 x 1/10GBASE-T RJ45

48 x SFP/SFP+ (1/10G)

Oversubscription

1:4

1:4

1:4

1:3

FCoE

Yes

No

Yes (30m / Cat6a/7)

Yes

Buffer

10.2MB

10.2MB

10.2MB

32MB

No

No

No

Yes

Uplink Ports

Shared Buffer

BRKARC-3452


Cisco Public

20

Nexus 2000 Fabric Extender Scalability • Nexus 5600 and 6000 support all types of FEX, including HP, Fujitsu, Dell, IBM FEX for blade servers. Adapter-FEX and VM-FEX are also supported.

Cisco Nexus 2000 FEX

48xFEX

Cisco Nexus 5600/6000

+ Nexus 5000

BRKARC-3452

Nexus 5500

Nexus 5600

Nexus 6004

Nexus 7000

Nexus 7700

L2

12

24

24

48

48

64

L3

x

16

24

24

48

64


Cisco Public

21



Cisco Public

22

Nexus 5672 - 72 x 10G Unicast Throughput 100

90 80 70 60

Throughput (% of line rate)

50 40 30 20 10 0

RFC 2544 Unicast L2 Port Pair RFC 2544 Unicast L3 Port Pair

64 100 100

80 100 100

96 100 100

128 100 100

256 100 100

512 100 100

1000 100 100

1024 100 100

Packet Size (Bytes)

BRKARC-3452


Cisco Public

23

1280 100 100

1518 100 100

2240 100 100

4096 100 100

9100 100 100

Nexus 5672 - 72 x 10G Multicast Throughput 100 90 80 70


60 50 40 30 20 10 0

RFC 3918 Mcast L2 One to Many Accumulated RFC 3918 Mcast L3 One to Many Accumulated RFC 3918 Mcast L2 One to Many Distributed RFC 3918 Mcast L3 One to Many Distributed

64 100 100 100 100

80 100 100 100 100

96 100 100 100 100

128 100 100 100 100

256 100 100 100 100

512 100 100 100 100

1000 100 100 100 100

1024 100 100 100 100

1280 100 100 100 100

Packet Size (Bytes)

BRKARC-3452


Cisco Public

24

1518 100 100 100 100

2240 100 100 100 100

4096 100 100 100 100

9100 100 100 100 100

Nexus 5672 – 72 x 10G Unicast 100% Load Latency 1600 1400 1200 1000 Average Latency (Nano Seconds)

800 600 400 200 0

RFC 2544 unicast L2 port pair RFC 2544 unicast L3 port pair

64 1001 937

128 1021 963

256 1091 1029

512 1111 1048

1024 1105 1047

1280 1104 1046

Packet Size (Bytes)

BRKARC-3452


Cisco Public

25

1518 1104 1045

2240 1103 1044

4096 1101 1043

9216 1098 1040

Nexus 6004 Performance – Fully Loaded Switch

BRKARC-3452


Cisco Public

26

Nexus 6004 Performance – Spirent Traffic Generator

BRKARC-3452


Cisco Public

27

Nexus 6004 - 96 x 40G Unicast Throughput 100 90 80 70 60


50 40 30 20 10 0

RFC 2544 Unicast L2 Port Pair RFC 2544 Unicast L3 Port Pair

64 100 100

80 100 100

96 100 100

128 100 100

256 100 100

512 100 100

1000 100 100

1024 100 100

1280 100 100

1518 100 100

Packet Size (Bytes)

Spirent third party performance report for Nexus 6004 BRKARC-3452


Cisco Public

28

2240 100 100

4096 100 100

9100 100 100

Nexus 6004 - 96 x 40G Multicast Throughput


100 90 80 70 60 50 40 30 20 10 0

RFC 3918 Mcast L2 One to Many Accumulated RFC 3918 Mcast L3 One to Many Accumulated RFC 3918 Mcast L2 One to Many Distributed RFC 3918 Mcast L3 One to Many Distributed

64 100 100 100 100

80 100 100 100 100

96 100 100 100 100

128 100 100 100 100

256 100 100 100 100

512 100 100 100 100

1000 100 100 100 100

1024 100 100 100 100

1280 100 100 100 100

1518 100 100 100 100

Packet Size (Bytes)

Spirent third party performance report for Nexus 6004 BRKARC-3452


Cisco Public

29

2240 100 100 100 100

4096 100 100 100 100

9100 100 100 100 100

Nexus 6004 - 384 x 10G Unicast 100% Load Latency 1.4 1.2

Average Latency1 (microseconds) 0.8 0.6 0.4 0.2 0 RFC 2544 Unicast L2 Port Pair RFC 2544 Unicast L3 Port Pair

64

80

96

128

256

512

1000

1024

1280

1518

2240

4096

9100

1.028

0.99

1.005

1.023

1.121

1.118

1.111

1.116

1.11

1.107

1.11

1.108

1.101

1.032

1.001

1.011

1.03

1.122

1.121

1.116

1.118

1.113

1.111

1.112

1.11

1.106

Packet Size (Bytes) BRKARC-3452


Cisco Public

30

Nexus 6004 - 384 x 10G Multicast 100% Load Latency 1.4 1.2 1

Average Latency (microseconds)

0.8 0.6 0.4 0.2 0

RFC 3918 Mcast L2 One-to-Many Accumulated RFC 3918 Mcast L3 One-to-Many Accumulated RFC 3918 Mcast L2 One-to-Many Distributed RFC 3918 Mcast L2 One-to-Many Distributed

BRKARC-3452

64

80

96

128

256

512

1000 1024 1280 1518 2240 4096 9100

1.004 0.989 1.029 1.017 1.095 1.086 1.079 1.084 1.083 1.082 1.082 1.08 1.077 1.067 1.051 1.061 1.064 1.087 1.083 1.079 1.081 1.08 1.079 1.078 1.076 1.073 0.951 0.949 0.954 0.962 1.019 1.042 1.043 1.042 1.042 1.042 1.043 1.043 1.043 0.96 0.958 0.963 0.974 1.019 1.044 1.044 1.044 1.044 1.044 1.044 1.044 1.044


Cisco Public

31

Nexus 6004 - 384 x 10G Unicast and Multicast 100% Load Jitter 12 10 8

Average Jitter (nanoseconds)

6 4 2 0

Unicast L2 Jitter (ns) Unicast L3 Jitter (ns) Multicast L2 Jitter (ns) Multicast L3 Jitter (ns)

64 8 8 9 9

80 5 5 6 5

96 8 9 9 9

128 5 5 5 5

256 6 6 5 5

512 8 8 8 8

1000 6 6 6 6

1024 7 7 7 7

Packet Size (Bytes)

BRKARC-3452


Cisco Public

32

1280 6 6 6 6

1518 10 10 10 9

2240 10 10 9 9

4096 9 9 9 9

9100 9 10 9 9



Cisco Public

33

Cisco Nexus 5672 Internal Architecture Switch Fabric Unified Fabric Controller (UFC)

Supervisor UPC 1

UPC 2

UPC 3

UPC 4

UPC 5

UPC 6

UPC 0

FC PHY

48xSFP+

BRKARC-3452


6xQSFP+

Cisco Public

34

CPU

Cisco Nexus 56128 Internal Architecture Slot 2

Slot 1

2x QSFP+

24xSFP+ FC PHY

FC PHY

UPC 7

UPC 8

UPC 9

2x QSFP+

24xSFP+ FC PHY

FC PHY

UPC 10

UPC 11

UPC 12

Supervisor Unified Fabric Controller (UFC)

UPC 1

UPC 2

UPC 3

UPC 4

48xSFP+ BRKARC-3452


Cisco Public

35

UPC 5

UPC 6

2x QSFP+

2x QSFP+

CPU

Cisco Nexus 6004 Internal Architecture Switch Fabric UFC 1

UPC 1

UFC 2

UPC 2

UPC 3

UFC 3

UFC 4

UPC 4

UPC 1

UPC 2

UPC 0

FC PHY

20xSFP+

12xQSFP+

CPU Supervisor

12x40G LEM (M12Q) BRKARC-3452


20x10G LEM (M20UP) Cisco Public

36

Unified Port Controller - UPC ASIC • Multimode MAC; built-in PHY for 1, 10, and 40GE

• Packet parsing and rewriting • Lookup engine and access control: L2, L3, FabricPath, ACL, FCoE, and policing • Buffering and queuing: Buffer management, PFC for lossless traffic, queuing (Strict Priority Queuing and DWRR), and packet replication (SPAN and multicast) • Extra fabric bandwidth for SPAN and multi-destination traffic

Front Panel Ports Twelve 10 GE

Fabric Connections 224Gbps UPC 448Gbps

224Gbps

Three 40 GE BRKARC-3452

UPC


448Gbps Cisco Public

37

Cisco Nexus 5600/6000 Packet Processing Flow Ingress Pipeline

VLAN Table MAC Table ARP/ND Table IPv4/IPv6 Routing Table FabricPath FC routing Table

Ingress UPC  Buffer Allocation  Buffer Accounting  PAUSE Signal

1, 10, and 40 GE

Unicast VOQ (8 per egress port)

Forwarding Lookup MAC

Packet Rewrite

Parser Ingress ACL Ingress Policing

Buffer Management

Buffer      BRKARC-3452

Security ACL QoS ACL SPAN Filter Policing/Stats Storm Control

Ingress SPAN Replication


Cisco Public

38

8000 Multicast VOQs

Unified Crossbar Fabric

     

Nexus 5600/6000 Packet Processing Flow Egress Pipeline


Egress UPC  Unicast buffer  Multicast congestion management

Unicast Queue

Buffer Management

Scheduling

Multicast Replication

Egress ACL Egress Policing*

Editing

TX SPAN Replication

Buffer Multicast Queue

To Ingress Buffer Management

*Egress policing is not supported with current software BRKARC-3452


Cisco Public

39

MAC

Cisco Nexus 5600/6000 Key Forwarding Tables  Host table: 256,000 -entry hashing table; actual capacity is slightly less than 256,000



Host table default carving: 128,000 MAC, 128,000 IP hosts

 Host table: Shared between MAC, ARP, and ND and /32 host route



LPM table: 32,000 entries. Also known as summary routes

256,000 Entries

Host Table*

MAC Region

LPM Table(32,000)

Summary Routes

IP Host Region (ARP/ND/Host route/(*,G)) * Hardware table size. Please check configuration limit for software scaling BRKARC-3452


Cisco Public

40

Mroute Table(64,000)*

(S,G)

Cisco Nexus 5600/6000 Host Table Carving • In Layer 2, virtualized environments, it can be interesting to have more than 128k MAC addresses on the switch • MAC/ARP Hardware Resource Carving allows to change the size of MAC and IP regions to allocate more space for the MAC region.

256,000 Entries

Host Table

MAC Region

IP Host Region (ARP & ND & /32 Host route)

BRKARC-3452


Cisco Public

41

Cisco Nexus 5600/6000 Carving Profiles • Use of pre-defined carving profiles. • HRT = Host Route Table = IP host region. • STM = Station Table Management = MAC region. Template Profile

Description

hrt-128-stm-128

HRT size: 128k, STM size: 128k (default profile)

hrt-96-stm-160

HRT size: 96k, STM size: 160k

hrt-64-stm-192


hrt-32-stm-224


BRKARC-3452


Cisco Public

42

Cisco Nexus 5600/6000 MAC/ARP Resource Carving CLI • Specify the resource template to use: •

•

switch(config)# hardware profile route resource servicetemplate template-name Save the config and reload the switch

• Show commands: • • • •

BRKARC-3452

show show show show

hardware profile route resource hardware profile route resource running-config hardware profile startup-config hardware profile


Cisco Public

43

template template default route resource template route resource template



Cisco Public

44

QSFP 40G Bidirectional Transceiver 40G deployment with 10G cabling • QSFP pluggable 40Gbit/s Bidirectional (BiDi) Transceiver that works with existing Multi-mode fiber infrastructure – Using the existing 10Gbit/s Multi-mode infrastructure – QSFP-40G-SR-BD has dual LC Connector – 40Gbit/s BiDi transceiver has two 20Gbit/s channels each transmitted and received simultaneously on two wavelengths

BRKARC-3452


Cisco Public

45

QSFP-40G-SR-BD  Supports 100m with OM3 with MMF and 150m with OM4 (OM4+) MMF 40G-BD

40G-BD Multimode LC Duplex

 LC duplex connectors  Wavelength: ~850nm and ~900nm  Pull-Tab Color: grey

Optical DeMux

40G-BD

10G-SR 10G-SR 10G-SR 10G-SR

Multimode LC Duplex BRKARC-3452


Cisco Public

46

QSFP-40G-SR4

40G-SR4

40G-SR4 MPO-12 MPO-12

• Support 100M with OM3 MMF and 150M with OM4 MMF 10G-SR 10G-SR 10G-SR 10G-SR

40G-SR4

MPO-12  LC Breakout BRKARC-3452


Cisco Public

47

QSFP-40G-CSR4 • Supports 300M with OM3 MMF and 400M with OM4 MMF 40G-CSR4

40G-SR4/CSR4

• Compatible with 10G SFP-SR

MPO-12 MPO-12

10G-SR 10G-SR 10G-SR 10G-SR

40G-CSR4

MPO-12  LC Breakout BRKARC-3452


Cisco Public

• Compatible with Cisco QSFP40G-SR4

48

Multimode Ribbon Fiber 40GE

For QSFP-40G-SR4 and QSFP-40G-CSR4 12-Fiber MPO Connector • MPO/MTP connector with 12 pins

• Use 4 fiber pairs − 4 TX and 4 RX allows for 40GE − 4 unused fibers in the center

BRKARC-3452


Cisco Public

49

QSFP-40GE-LR4 • QSFP-40GE-LR4 can only be connected to each other. 10KM with Single mode fiber. 4x ROSA

Multiplexer LC

TIA

Host Card

IEEE Standard 40GBASE-LR4

4x TOSA LC

Can’t connect to 10Gbase-LR

BRKARC-3452


Cisco Public

50

FET-40G • • • •

Low-cost QSFP optical transceiver connecting FEX to Cisco Nexus 6004 Supported on Cisco Nexus 5600/6004 and Nexus 2248PQ-10G Interoperable with FET-10G Support for 100m distance with OM3 Cisco Nexus 5600/6000

Cisco Nexus 5600/6000 FET-40G

FET-40G

FET-10G FET-40G Cisco Nexus 2248PQ-10G BRKARC-3452


Cisco Public

51

Cisco Nexus 2232PP and 2232TM-E 2232TM Cisco Nexus 2248TP-E 2248TP



Cisco Public

52

ACL Types and Features • Security ACL − MAC, IPv4, and IPv6 ACLs − PACL: ACL enabled under L2 interface − VACL: ACL enabled for L2 VLAN traffic − RACL: ACL enabled for routed traffic − RBACL: Role-based ACL for CTS (roadmap feature)

• ACL for QoS classification and Policing • Policy Based Routing (PBR) – User-configured ACL to redirect traffic

• ACL for SPAN/ERSPAN • ACL for control traffic – To redirect control traffic to CPU, not user-configurable – CoPP BRKARC-3452


Cisco Public

53

ACL Scaling and TCAM Partition • 4096 Access Control Entries per UPC are organized in blocks of 64 ACE entries. • Each IPv6 ACL (without port range) requires two TCAM entries. Default TCAM Partition

VACL(1024) 4096 Entries

IFACL(1152) QoS(448) RBACL(1024) SPAN(64) Control Traffic(256) BRKARC-3452

switch# sh platform afm info tcam 0 region ifacl ifacl tcam TCAM configuration for asic id 0: [ vacl tcam]: range 0 - 1023 [ifacl tcam]: range 1024 - 2175 * [ qos tcam]: range 2176 - 2623 [rbacl tcam]: range 2624 - 3775 [ span tcam]: range 3776 - 3839 [ sup tcam]: range 3840 - 4095

TCAM [ifacl tcam]: [v:1, size:1152, start:1024 end:2175] TCAM Region In use tcam entries: 24 Usage 1024-1039,2168-2175


Cisco Public

54

TCAM Carving • TCAM Carving allows to change the size of each region • Create a template: switch(config)# hardware profile tcam resource template t1 switch(config-tcam-templ)#

• Set region sizes: switch(config-tcam-templ)# ifacl ? <320-3584> Enter size of ifacl increments)

BRKARC-3452


Cisco Public

55

region

(in

64

entry

TCAM Carving • Similarly, other region sizes can be changed: switch(config-tcam-templ)# ? ifacl Configure size of ifacl region no Change size to default value qos Configure size of qos region rbacl Configure size of rbacl region span Configure size of span region vacl Configure size of vacl region

• Display the configuration for a template along with the current usage: switch(config)# show hardware profile tcam resource template name

BRKARC-3452


Cisco Public

56

TCAM Carving switch(config-tcam-templ)# show hardware profile tcam resource template name t1 ------------------------------------------------------------------Template name: t1 Current state: Created Region Size-allocated Current-size Current-usage Available/free ------------------------------------------------------------------Vacl 1152 1024 28 996 Ifacl 1024 1152 266 971 Rbacl 1152 1152 3 1149 Qos 448 448 30 418 Span 64 64 6 58 Sup 256 256 74 182 -------------------------------------------------------------------

BRKARC-3452


Cisco Public

57

TCAM Carving • Commit a template: switch(config)# hardware profile tcam resource service-template t1 Details of the t1 template you are trying to commit are as follows: ------------------------------------------------------------------------------Template name: t1 Current state: Committed (Startup/Running) Region

Features

Size-allocated

Current-size

Current-usage

Available/free

------------------------------------------------------------------------------Vacl

Vacl

1152

2048

11

2037

[…] ------------------------------------------------------------------------------

Template committed. Please do the following for the template to be applied: 1> Save running config :

"copy running-config startup-config"

2> Reboot the switch

"reload”

BRKARC-3452

:


Cisco Public

58



Cisco Public

59

Why Analytics? – A Data Center Today: • 1G -> 10G -> 40G -> 100G

• Virtualized • More and more applications • Those applications are more and more complex (Hadoop…)

• More and more flows • Latency is very important • Ports speed mismatch • Buffer discrepancy between the devices • With spine/leaf, design is less complex, but there’s more speed and more applications BRKARC-3452


Cisco Public

60

Switched Port Analyzer (SPAN) Switch

Host-B

Host-A

Monitor Destination

Monitor Source

Sniffer Device

BRKARC-3452


Cisco Public

61

Encapsulated Remote SPAN (ERSPAN) Host-B

GRE Encapsulated monitored traffic

IP Cloud Host-A

Monitor Destination

Monitor Source

Sniffer Device BRKARC-3452


Cisco Public

62

Cisco Nexus 5600/6000 SPAN SPAN Features

Nexus 5600/6000

Nexus 5500

Total SPAN sessions

31 bi-directional (16 with current NXOS)

4 bi-directional

Local SPAN sessions

31 bi-directional (16 with current NXOS)

4 bi-directional

ERSPAN sessions Prioritize data over SPAN

16 Yes (through scheduling)

4 Yes (SPAN policing)

Line-rate SPAN throughput

Yes

No

ERSPAN destination session ERSPAN with 1588 PTP timestamp

Yes Yes*

No No

MTU Truncated SPAN and ERSPAN ACL filter for SPAN and ERSPAN

Yes Yes

Yes Yes

SPAN-on-Drop SPAN-on-Latency

Yes Yes

No No

SPAN with multiple destination ports

Yes (each destination port uses one SPAN session)

No

*Software roadmap feature BRKARC-3452


Cisco Public

63

High-Performance SPAN Extra Fabric Bandwidth for SPAN

Scheduler Prioritizes Production Data

Data


SPAN Separate Buffer Pool for SPAN

BRKARC-3452


Cisco Public

64

SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014

SPAN-On-Drop

Tail-Drop

SPAN-on-Drop

BRKARC-3452

Ingress Data Buffer

Monitoring Station

SPAN Buffer


Cisco Public

65

SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014 SPAN-on-Drop • Copies the dropped packet to capture device • Correlate the packet drop with application • Works for unicast packets only • Packet drops can be monitored on ingress only • There is a dedicated buffer space for SPAN, so SPAN-on-Drop does not affect production traffic • Supports both local SPAN and ERSPAN • Configuration uses existing SPAN commands • One SPAN-on-Drop session is supported • Can have multiple source ports, and multiple destination ports • Source cannot be FEX HIF port. But FEX fabric port is supported • Source port(s) can be a part of a SPAN-on-Drop session, and a local SPAN session simultaneously BRKARC-3452


Cisco Public

66

SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014

SPAN-on-Drop

The source interface is the ingress port for which we want to monitor drops.

switch(config)# monitor session 1 type span-on-drop

switch(config-span-on-drop)# source interface e3/1 switch(config-span-on-drop)# destination interface e3/2

switch(config)# monitor session 2 type span-on-drop-erspan switch(config-span-on-drop-erspan)# source interface e3/1 switch(config-span-on-drop-erspan)# destination ip 100.1.1.2

BRKARC-3452


Cisco Public

67

Latency Monitoring Latency Monitoring: Min: 856ns Max: 1208ns Avg: 901ns

T0 T1 – T0

T0 T1 – T0

• •

TimeStamp Packet

BRKARC-3452


Cisco Public

Latency Histogram: Latency Range: 800ns - 10usec Stats: Within the Range: x packets Out of the Range: y packets

68

FIFO Latency Per Port-Pair Data

Latency LatencyMonitoring Monitoring - NX-OS 7.0(1)N1(1), Q1/2014 • Latency Monitoring provides {min, average, max} latency between a specified port pair and also maintains latency histogram (accuracy in few nanosecs) • By default instantaneous Latency Monitoring is enabled between pair of ports • Latency Histogram can be enabled for specific port-pair to provide histogram instead of instantaneous mode • Measures switch latency for each packet, no sampling required • Fully implemented in HW, no CPU impact, no traffic impact

BRKARC-3452


Cisco Public

69

Instantaneous Mode • Enabled by default on all pairs of ports. No configuration is required switch# show hardware profile latency monitor interface e1/23 interface e1/22 -------------------------------------------------------------------------------Latency Statistics in nano seconds -------------------------------------------------------------------------------| Ingress Port | Egress Port | Minimum | Maximum | Average | -------------------------------------------------------------------------------| Ethernet1/22| Ethernet1/23| 856| 1208| 901| --------------------------------------------------------------------------------

BRKARC-3452


Cisco Public

70

Custom Histogram • Count the number of packets falling in a specific range of latency • Example for port-pair e1/1 and e1/3: switch(config)# interface e1/3 switch(config-if)# packet latency low-latency 800 high-latency 10000 mode custom interface e1/1

switch# show hardware profile latency monitor histogram interface e1/3 interface e1/1 -------------------------------------------------------------------------------| Egress Interface : Ethernet1/3 Ingress Interface : Ethernet1/1 | -------------------------------------------------------------------------------| All Latency Values are in nano seconds | -------------------------------------------------------------------------------| Range | 800 <= Latency < 10000 | Outside the first range | ------------------------------------------------------------------------------| counter | 3542818572 | 16290988 |

BRKARC-3452


Cisco Public

71

SPAN-on-Latency Latency Monitoring: Min: 856ns Max: 1208ns Avg: 901ns

T0 T1 – T0

T2 T3 – T2

TimeStamp Packet

BRKARC-3452


Cisco Public

If Latency Threshold > 10 usec: SPAN to 1/64

Monitoring Station

72

SPAN-on-Latency • This feature replicates packets from the interface when latency exceeds a configured threshold • Latency threshold is per-port • This again helps to identify the victim of congestion • Replicated traffic uses the SPAN buffer so it doesn't impact the production traffic • One SPAN-on-Latency session is supported in hardware • Uses similar SPAN CLI, with new session type “span-on-latency” • ERSPAN-on-latency is also supported BRKARC-3452


Cisco Public

73

SPAN-on-Latency Configuration • The SPAN-on-Latency source port is the egress port on which we monitor the latency • The SPAN-on-Latency session makes a copy of all high-latency packets egressing on this port, coming from any ingress port • Sample configuration to SPAN packets if latency goes beyond 10us when egressing on e1/3: Always Tx: packets egressing on 1/3 (any source) with latency >10us will be replicated to the SPAN dest 1/4

monitor session 1 type span-on-latency source interface Ethernet1/3 tx destination interface Ethernet1/4 interface Ethernet1/3 hardware profile latency monitor threshold 10001 interface Ethernet1/4 switchport mode monitor BRKARC-3452


Cisco Public

74

SPAN-on-Latency • Source port can be an regular ethernet port, not a port-channel. Can be a port-channel member

• Source port cannot be FEX HIF port. But FEX fabric port is supported • Multiple sources can be configured – latency threshold is per SPAN-ondrop TX source port

• A SPAN-on-Latency source port cannot be in another SPAN session • Destination is only a single ethernet port, not port-channel • Only one destination port can be configured

BRKARC-3452


Cisco Public

75



Cisco Public

76

Cisco Nexus 5600/6000 Multicast Features Summary • IGMP snooping for IGMPv1, v2, and v3 • PIM-SM • PIM-SSM without vPC and PIM-SSM with vPC+ • PIM-BiDir without vPC and PIM-BiDir with vPC+ • PIM Policy • MSDP • Anycast RP with PIM (RFC 4610) • Anycast RP with MSDP • 8000 IGMP snooping entries

• 8000 mroutes with vPC, 16000 mroutes without vPC supported by software BRKARC-3452


Cisco Public

77

Efficient Multicast Replication • Optimized multicast replication • Fabric replication and egress replication; one copy is replicated to egress UPC, where there is a receiver—minimizing the traffic load on the switch fabric and eliminating the switch fabric congestion

• Line-rate multicast replication Egress UPC

Ingress UPC

BRKARC-3452



Egress UPC

Cisco Public

78

Multicast VOQ • 8000 multicast VOQs to eliminate HOLB and help ensure high throughput • Tracks the fan-out of the egress UPC; packets with different egress UPC fanouts are assigned to different VOQs so that it can be scheduled and served independently Ingress UPC

Multicast VOQ Unicast VOQ

Egress UPC 1 224.1.1.1


Egress UPC 2

224.1.1.1

224.1.1.2

Egress UPC 3 224.1.1.2

224.1.1.2 BRKARC-3452


Cisco Public

79

Cisco Nexus 5600/6000 Key Forwarding Tables  Host table: 256,000 -entry hashing table; actual capacity is slightly less than 256,000



Host table default carving: 128,000 MAC, 128,000 IP hosts

 Host table: Shared between MAC, ARP, and ND and /32 host route



LPM table: 32,000 entries. Also known as summary routes

256,000 Entries

Host Table*

MAC Region

• Mroute table: 64,000 entries

LPM Table(32,000)

Summary Routes

IP Host Region (ARP/ND/Host route/(*,G)) * Hardware table size. Please check configuration limit for software scaling BRKARC-3452


Cisco Public

80

Mroute Table(64,000)*

(S,G)

MET Table • 64,000 entries in MET table • Currently supported: 8,000 vPC, 16,000 non-vPC

Ingress UPC

Multicast VOQ Unicast VOQ

• Today the MET table is the limiting factor for multicast scale MET: 123

4

16384 16385

122 123

MET: 123

1

1, 4, 5

5

65536

Egress UPC 1 2, 10

2 MET: 123

16384 16385 10

Switch Fabric BRKARC-3452

122 123


Cisco Public

65536 81

Egress UPC 2

Multicast Hashing over Port Channel • N5600/6000 implements flow based hashing for multi-destination traffic, and it supports multicast load sharing over Port Channel with 5-tuple packet header.

• Traffic is replicated to all egress UPCs where Port Channel member resides • Egress UPC runs hash calculation and one egress port is chosen to send out multicast packets. The UPC ASIC that is not supposed to send out packet will drop packet (Egress UPC 1 in the example)

Multicast VOQ

Ingress UPC

Unicast VOQ

Egress UPC 1

BRKARC-3452


Port 3 Selected


Port 1 Port 2

Egress UPC 2 Port 3 Port 3 Selected Port 4

Cisco Public

82



Cisco Public

83

Cisco Nexus 5600/6000 QoS Features • Eight classes of service; – 2 reserved for control traffic, 6 for data traffic

• Traffic classification – DSCP, CoS, and ACL

• Strict Priority Queuing and DWRR – DCBX 802.1Qaz

• Packet marking – DSCP, CoS, and ECN

• Ingress and egress policing – 4096 policers per ASIC

• No drop system class • Flexible buffer management BRKARC-3452


Cisco Public

84

Cisco Nexus 5600/6000 QoS Processing Flow Ingress UPC Trust CoS/DSCP L2/L3/L4 info with ACL MAC

Traffic Classification

Ingress Policing

MTU Checking

Per-class Buffer Usage Monitoring

VOQs for Unicast (8 per egress port)

Multicast Queues

Truncate or Drop Packets if MTU is Violated

Egress Queues

PAUSE ON/OFF Signal MAC

Unicast

ECN Marking

Egress Policing

Egress Scheduling Multicast

Egress UPC BRKARC-3452


Cisco Public

Strict Priority + DWRR Scheduling 85

Crossbar Fabric

Ingress Cos/DSCP Marking

If Buffer Usage Crosses Threshold: • Tail drop for drop class • Assert pause signal to MAC for no-drop system class

Increased Packet Buffer • 25MB packet buffer is shared by every three 40GE ports or twelve 10GE ports. 150MB of total buffer

• Nexus 56128:

300MB of total buffer

• Nexus 6004:

800MB of total buffer

Ingress UPC

Multicast VOQ

16MB

Unicast VOQ

• Nexus 5672:

BRKARC-3452


Egress UPC

9MB Unified Crossbar Fabric

Cisco Public

86

Ingress Buffer - Flexible Buffer Management • Shared buffer is good for burst absorption.

UPC Ingress Buffer (16MB)

• Dedicated buffer is good for predictable performance for each port.

SPAN

• N5600/6000 buffer management is flexible: dedicated plus shared.

Shared Packet Buffer ~14MB

• Long-distance FCoE, video editing (i.e., AVID), Big Data, and distributed storage

BRKARC-3452


Cisco Public

Control

Port 1 Dedicated

87

Port 2 Dedicated

Port 3 Dedicated

Tune Shared vs Dedicated Buffer Allocation at Ingress • “queue-limit” under “network-qos” policy specifies the dedicated buffer for each port and each class. • Without “queue-limit” each class of service will get 100 KB of dedicated buffer. • The size of dedicated buffer can be different for different classes of service. The policy applies to all ports in the chassis. • Total ingress buffer minus the dedicated buffer and buffer for control and SPAN will be in the shared buffer pool.

• The following example sets the dedicated buffer for “class-default” to be 400 KB for all ports. switch(config)# policy-map type network-qos Policy-buffer switch(config-pmap-nq)# class type network-qos class-default switch(config-pmap-nq-c)# queue-limit 400000 bytes switch(config-pmap-nq-c)# system qos switch(config-sys-qos)# service-policy type network-qos Policy-buffer BRKARC-3452


Cisco Public

88

Egress Buffer Design • 9-MB buffer is reserved for egress traffic

UPC Egress Buffer (9MB)

• Unicast traffic can be buffered at egress and ingress.

Unicast Buffer

• Multicast is buffered at egress in case of interface oversubscription.

Dedicated per port

Multicast Buffer Shared ~6MB

BRKARC-3452


Cisco Public

89

Unicast Flow Control and Buffering • Unicast is queued at egress buffer pool first and then ingress buffer after egress queue is full. • Take advantage of ingress buffer from multiple port or ASIC for unicast burst absorption. • Ensure fairness among multiple ingress ports

BRKARC-3452


Cisco Public

90

Identify Packet Drops Due to Congestion switch# show interface ethernet 1/7

switch# show queuing interface ethernet 1/7 Ethernet1/1 queuing information: TX Queuing qos-group sched-type oper-bandwidth 0 WRR 100

Ethernet1/1 is down (SFP not inserted) Dedicated Interface

RX 0 input error 0 short frame 0 overrun 0 underrun 0 ignored 0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop 0 input with dribble 0 Rx pause

TX

RX Queuing qos-group 0

q-size: 100160, HW MTU: 1500 (1500 configured) drop-type: drop, xon: 0, xoff: 0 Statistics: Pkts received over the port : 710025080 Ucast pkts sent to the cross-bar : 474177592 Mcast pkts sent to the cross-bar :0 Ucast pkts received from the cross-bar : 0 Pkts sent to the port :0 Pkts discarded on ingress :

235847488 input discard Unicast drop at ingress

0 unicast packets 0 multicast packets 0 broadcast packets 0 output packets 0 bytes 0 jumbo packets 0 output error 0 collision 0 deferred 0 late collision

235847488

Per-priority-pause status (Inactive), Tx (Inactive)

0 lost carrier 0 no carrier 0 babble 0 output discard 0 Tx pause Broadcast/Multicast/Unknown

Ingress drop for each class

unicast flooding drop at egress BRKARC-3452


Cisco Public

Dedicated buffer for this class

91

: Rx

Identify Congested Egress Port with VOQ Counters I X I A I X I A

1/7

Ingress UPC 3

Unified Crossbar Fabric Egress UPC 3

1/8 Ingress UPC 4

1/10

 Using “show interface” CLI, we noticed drops on ingress ports  How do we know which egress port is affected? BRKARC-3452


Cisco Public

92

I X I A

Step 1: Map Interface to UPC Number N6004# show hardware internal bigsur all-ports Bigsur Port Info: Port |asic|inst|inst| name |idx |slot|asic|eport|logi|flag|adm|opr|if_index|diag|ucVer ---------+----+----+----+-----+----+----+---+---+--------+----+----sup1 |0 |17 |0 | 0 - |0 |b3 |en |dn |15010000|pass| 0.00 sup0 |0 |17 |0 | 1 - |1 |b3 |en |dn |15020000|pass| 0.00 40gb1/1 |1 |0 |0 | 0 - |0 |b3 |en |dn |1a070000|pass| 0.00 40gb1/2 |1 |0 |0 | 1 - |1 |b3 |en |dn |1a071000|pass| 0.00 40gb1/3 |1 |0 |0 | 2 - |2 |b3 |en |dn |1a072000|pass| 0.00 40gb1/4 |2 |0 |1 | 0 - |3 |b3 |en |dn |1a073000|pass| 0.00 40gb1/5 |2 |0 |1 | 1 - |4 |b3 |dis|dn |1a074000|pass| 0.00 40gb1/6 |2 |0 |1 | 2 - |5 |b3 |dis|dn |1a075000|pass| 0.00 40gb1/7 |3 |0 |2 | 0 p |6 |b3 |en |up |1a076000|pass| 0.00 40gb1/8 |3 |0 |2 | 1 p |7 |b3 |en |up |1a077000|pass| 0.00 40gb1/9 |3 |0 |2 | 2 p |8 |b3 |en |up |1a078000|pass| 0.00 40gb1/10 |4 |0 |3 | 0 p |9 |b3 |en |up |1a079000|pass| 0.00 40gb1/11 |4 |0 |3 | 1 - |10 |b3 |dis|dn |1a07a000|pass| 0.00 40gb1/12 |4 |0 |3 | 2 p |11 |b3 |en |up |1a07b000|pass| 0.00 Done. BRKARC-3452


Cisco Public

93

Step 2: Identify Congested Egress Port with VOQ Counters N6004# show platform software qd info counters voq asic-num 3 +----------+------------------------------+------------------------+-----------+ | port| TRANSMIT| TAIL DROP| HEAD DROP| +----------+------------------------------+------------------------+-----------+ Eth1/8 QUEUE-3 3888954496 2248133231 0 +----------+------------------------------+------------------------+-----------+

Congested egress port and class of service N6004# show platform software qd info counters voq asic-num 4 +----------+------------------------------+------------------------+-----------+ | port| TRANSMIT| TAIL DROP| HEAD DROP| +----------+------------------------------+------------------------+-----------+ Eth1/8 QUEUE-3 2574819434 2248180087 0 +----------+------------------------------+------------------------+-----------+

BRKARC-3452


Cisco Public

94

Ingress UPC number



Cisco Public

95

Nexus 5600/6000 Applications Compact Aggregation Nexus 7000

CORE L3

Nexus 7000 Nexus 6004

AGG. vPC to Nexus 7000 or 6004

Nexus 6004/ Nexus 5600

ACCESS

Nexus 2000

FEX

BRKARC-3452


Cisco Public

96

Nexus 5600/6000 Applications Large Scale Fabric (Layer 2 or Layer 3)

CORE

Nexus 7000

L3 SPINE

Nexus 6004

FabricPath Nexus 6004/ Nexus 5600+2000

BRKARC-3452


LEAF

Cisco Public

97

Nexus 5600/6000 Applications High Performance Computing (HPC)

CORE/AGG.

Nexus 6004

L3/L2 Fabric Nexus 3548 Nexus 5600/6004

BRKARC-3452


ACCESS

Cisco Public

98

Target Q3 CY14

Nexus 6004X Chassis • Nexus 6004X has 8 slots (similar to the 6004EF chassis) • Nexus 6004X chassis supports VXLAN functionality

• Provides Flexible Ordering Options – mix/match 10/40G LEMs • Supports new 12 x 40G LEM VXLAN capable (backward compatible with Nexus 6004EF chassis) • VXLAN capable LEM will work in non-VXLAN mode in 6004EF chassis • Same software as on Nexus 6004 • Supports all the optics, FEX models as on Nexus 6004

BRKARC-3452


Cisco Public

99

Nexus 5600/6000 Key Takeaways Rich Feature set

STP/vPC/FabricPath/DFA, FEX, L3, VXLAN, Adapter-FEX/VM-FEX

Performance

Line rate L2 and L3 with low latency

Scalability

256K MAC / 128K ARP / 32K LPM routes, 16K bridge domains with segment ID

Innovations

Intelligent SPAN, Buffer monitoring, Latency monitoring

BRKARC-3452


Cisco Public

100

Nexus 2000/5600/6000 More Information • http://cisco.com/go/nexus2000 • http://cisco.com/go/nexus5000 • http://cisco.com/go/nexus6000

BRKARC-3452


Cisco Public

101

Complete Your Online Session Evaluation • Give us your feedback and you could win fabulous prizes. Winners announced daily. • Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center. Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

BRKARC-3452


Cisco Public

102

Continue Your Education • Demos in the Cisco Campus • Walk-in Self-Paced Labs • Table Topics

• Meet the Engineer 1:1 meetings

BRKARC-3452


Cisco Public

103

Cisco Nexus 6000 and 5600 with Fabric Extender 2000 Switch

Recommend Documents