by Bryce Trapier | March 11, 2021 | Blog

Cisco Nexus Fabric Modules

a FABulous World

The Cisco Nexus 7000 chassis introduced the mass populous of Cisco networking consumers to the Fabric Module – or FAB’s as they’re commonly-called because of their original Stock Keeping Unit nomenclature (e.g. N7K-C7010-FAB-2).  The nomenclature for the Nexus 9500 chassis uses -FM (e.g. C9504-FM), which has led to some interesting nicknames, but all Cisco Nexus chassis-based switches (7000, 7700, and 9500 series) use Fabric Modules. 

Cisco Nexus 7000 FAB-2 Fabric Module

Cisco Nexus 9504 N9K-C9504-FM-E Fabric Module

You’ll likely hear the term FAB’s instead of FM’s when speaking about Fabric Modules with Cisco engineers and other networking professionals, so I’ll use the terms FAB and Fabric Module interchangeably for the rest of this article.

In the Cisco Nexus chassis-based architecture, the FAB’s provide backplane switching capacity for linecards in the switch.  The Supervisor Engines and System Controllers are responsible for chassis management and intelligence, and each linecard has local switching intelligence, but crossbar connectivity – bandwidth availability for chassis linecards to communicate with each other – is handled by multiple Network Forwarding Engines (NFE’s) on the FAB’s. 

Kind of like we used to say about the commo team in the Army, “You can talk smack about FAB’s, but you can’t talk smack without FAB’s.”

Fabric Modules provide a couple of benefits over switch designs where the supervisor handles crossbar connectivity.

  • The Fabric Modules plug into the back of the chassis, and do not consume a linecard slot.
  • Supervisor engines don’t need to house the physical hardware for crossbar connectivity – so the supervisors can be physically smaller, and they can cost less.
  • You still pay the full monetary price for connectivity when you initially purchase the supervisors and the FAB’s, but you can upgrade them independently. Upgrading the Supervisor doesn’t necessarily dictate an upgrade to the FAB’s.  And if you need more backplane bandwidth, you don’t necessarily have to upgrade your Supervisor (Catalyst 6500 users, rejoice!).
  • Speaking of upgrades: Upgrading FAB’s in the rear of the chassis, and upgrading half-width supervisors in the front of the chassis, is Much. Easier than upgrading a full-width supervisor engine that has been swallowed by cabling in front of the chassis.
  • One more bonus: most of the Fabric Modules for the Cisco Nexus chassis are hot-swappable. Yes, in many scenarios, you can replace your chassis backplane without powering down your switch.  I’ll pause a moment while that sinks in.

Fabric Modules make the backplane of the switch user-serviceable.  So why do people dislike the Fabric Modules

I think it’s because some people don’t understand the Fabric Modules.  And many people don’t understand them, because they’re a headache to understand.  And most people, even stubborn engineers, don’t like headaches.

So let’s talk FAB’s without breaking out the headache medicine.

PLEASE NOTE: the diagrams and accompanying narratives in the rest of this article are going to simplify some things, to facilitate a high-level understanding.  As such, these are not going to be a 100% accurate representation of all the guts in the switch. 

The Problem

At a very high level, here’s how internal bandwidth on a pizza-box switch or on a linecard works: you solder on an ASIC that is rated for X-amount of bandwidth, and you connect a bunch of ports to that ASIC.  I’m leaving out about 800 important concepts and components here, but for the sake of simplicity let’s just go with “the front-side ports, that you and I can see and touch, connect to port ASICs.  And those port ASICs connect to some chassis backplane stuff that connects all of the port ASICs together.” 

As an example, here’s what a 48-port switch might look like with 10G port ASICs.  Again, please note that this block diagram is oversimplified to facilitate high-level understanding.

You can see a potential problem here – if all twelve of those access ports receive the maximum bandwidth, the port ASIC will be overrun.  Consider that some of those port ASICs go back to “neighborhood ASICs” that aggregate some of the port ASICs before attaching to the backplane.  This architecture might cause drops on the input, but the bottleneck at the port ASIC protects the rest of the backplane.  That’s why it’s done (and, it also keeps manufacturing costs down for the tradeoff of, “Will you ever really see all access ports transmitting at sustained maximum bandwidth?”)

Let’s apply that concept to a chassis-based switch, where each linecard’s port ASICs get aggregated to a neighborhood ASIC and eventually get aggregated to “a bandwidth number” that connects to the backplane.  Again, we are simplifying some concepts here.

A chassis backplane that provides 1.5 Tbps of bandwidth to each linecard slot works well for a chassis that’s populated with 32-port 40G linecards.  The fabric provides 1.5 Tbps at the backplane, and the linecards cards require 1.2 Tbps to run at line-rate.    But as soon as we throw in that 36-port 100G linecard, we overrun the backplane by more than 2x. 

The above scenario was quite frequent in the sophomore years of the Nexus 7000 series chassis, when network designers sometimes added linecards without considering the modular backplane (and, it was simply “the way” in most Catalyst 4500 and 6500 designs).  Sometimes the oversubscription scenario is a known entity – maybe the network designers know that only twelve of those 100G ports would be consumed.  But, oversubscription from the linecard to the backplane is a caveat that network designers should consider.

The Solution

Wouldn’t it be cool – especially in our “new linecard” scenario above – if we could just upgrade the fabric backplane?  Without upgrading switches.  Without upgrading the Supervisor Engine.  And, ideally, without powering-down the switch.  That’s exactly what the Fabric Modules allow us to do.  The Fabric Modules give us a modular backplane.

It’s worth restating at this point that each FAB on the Nexus 9500 platform houses multiple Network Forwarding Engines (NFE’s). 

Now is a good time to mention that the linecards in the chassis also house NFE’s (in addition to other ASICs on the linecards).  The NFE’s on the linecards talk to the NFE’s on the FAB’s.

Each NFE, on the FAB and on the linecards, has multiple traces.  The Gen1 FAB’s for the Nexus 9500 had two NFE’s, and each NFE had 32 traces of 40 Gbps.  The Gen1 FAB could present up to 8 of those traces – 320 Gbps – to each linecard slot.  Multiply that 320 Gbps by the 6 available Fabric Module slots in the Nexus 9500 chassis, and your backplane using the Nexus 9500 Gen1 FAB’s was 1.92 Tbps per slot (or 3.84 Tbps full-duplex “Cisco Math” if you were a Sup720 consumer).

To tie things together, consider that each trace on the NFE’s physically connect — somewhere.  That somewhere is the chassis midplane.  When a Fabric Module is inserted into the rear of the chassis, the NFE traces on the Fabric Modules physically touch a piece of metal in the receiver slot in the chassis.

When a linecard is inserted into the front of the chassis, the NFE traces on the linecard physically touch a piece of metal in the receiver slot.  Don’t lose sight of the concept that the linecards and the backplane – while both are modular – are still physically connected.

Let’s apply that concept to a chassis-based switch, where each linecard’s port ASICs get aggregated to a neighborhood ASIC and eventually get aggregated to “a bandwidth number” that connects to the backplane.  Again, we are simplifying some concepts here.

Visualizing the FAB’s and the linecards in the same chassis, gives us something like this (again, this is an oversimplified representation), where each of the blue lines represents a physical connection that is facilitated by the midplane of the chassis:

The strips of metal on the chassis midplane don’t care what the bandwidth is – they aren’t printed Ethernet like xGBASE-KR – they’re simply strips of metal that provide physical connectivity between the linecard and the FAB’s.  As long as the electricity passing between linecards and FAB’s doesn’t physically melt the metal connectors on the midplane, the chassis doesn’t care.  The FAB’s just need to keep up with the linecards. 

My final technical note before we get to the FAB specs, and it’s a word of caution, to design engineers who have made it this far in the article: all Fabric Modules in a given Nexus chassis must be the same Fabric Module type.  You’re upgrading the chassis backplane, which is a Clos architecture.  It needs to be consistent.  Good design engineers have a voice in our mind that pushes us to ask, “What if?”  Continue to ask that “What if,” but don’t mix FAB types in the same chassis.

At the time of writing, February 2021, these are the classes of Fabric Modules available for the Cisco Nexus 9500 switches:

  • FM: The Gen1 9500 FAB provided 320 Gbps bandwidth, per FAB, to each linecard slot. These have been announced End-of-Life and, like COVID masks without fancy designs, nobody should use these anymore.
  • FM-E: The FM-E FAB provides up to 800 Gbps capacity per line card slot – more than double the bandwidth of the FAB Gen1 modules. These FAB’s are only compatible with the Nexus 9504 chassis.
  • FM-E2: The FM-E2 FAB is the FM-E equivalent for the Nexus 9508 and the Nexus 9516 switches.  These FAB’s provide up to 800 Gbps capacity per line card slot.  The FM-E2 FAB is only compatible with the Nexus 9508 and the Nexus 9516 chassis.  In fact, at time of writing, the FM-E2 FAB’s are the only FAB’s orderable with the Nexus 9516 switch*  So, you’ll want to use a Nexus 9504 or Nexus 9508 switch if your deployment requires any other Fabric Module in this list.
  • FM-G: The FM-G “Cloud-Scale” FAB’s provide up to 1.6 Tbps capacity per line card slot (consider that the Gen1 FAB provided 320 Gps per slot). At the time of writing, the FM-G FAB’s only support ACI mode – they’ll support NXOS mode in Nexus code 10.1.  NXOS 10.1 should be available shortly after this article goes to print, but at time of writing this disclaimer is still relevant.
  • FM-R: The FM-R FAB’s are a special-purpose design that come with a caveat. They provide 900 Gbps per line card slot, but are only compatible with R-Type line cards.  R-Type line cards and FAB’s are for environments with deep buffer requirements – while the Cloud-Scale linecards provide a maximum of 160 MB of packet buffer, the R-Type linecards provide up to 24 GB of packet buffer on a QSFP28 linecard.  R-Type line cards and FMs are not suited for VXLAN or ACI deployments.
  • FM-S: At the time of writing, the FM-S FAB’s can be ordered, but these will go end of sale in August 2021. These FAB’s support 100G ports, but only in NXOS mode and are only compatible with one linecard – the X9432C-S.  The X9432C-S was a necessary linecard when it was introduced, but there are better options now for a number of reasons.  Like an undecorated COVID mask, just don’t entertain the FM-S FAB or the X9432C-S linecard.  If you’re looking at 100G ports, you should look at the other Fabric Modules.
    • * You might see an FM-E FAB on a Cisco Nexus 9516 chassis.  In days past, you could order an FM-E module with a 9516, but the flow sizes on the 9156 platform max out at 50 Gbps with the -E module and 100 Gbps with the -E2 module.  This and other reasons motivated Cisco to discontinue availability of the FM-E module on the 9516 platform.  The 9508 platform can handle 100 Gbps flows using either the FM-E or the FM-E2 modules.  Also, the FM-G FAB modules should be coming soon to the Nexus 9516 chassis.

So now you know more about Fabric Modules for the Cisco Nexus chassis-based switches.

Newer linecards are going to present higher-bandwidth NFE traces to the chassis.  The fabric backplane needs to keep up.  A modular backplane makes that easier, but don’t let the modularity make you complacent.  It’s always a good idea to evaluate backplane requirements and interoperability support against any linecards that you plan to install in your chassis-based switch. 

Decoupling the supervisor, from the front-side port ASICs and from the fabric backplane, provides a lot of design flexibility.  But with great flexibility comes great lists of caveats. 

ABOUT THE AUTHOR

Bryce Trapier is a Technical Solutions Architect at PEAK Resources, Inc. in Denver, Colorado.  He has over 20 years’ experience designing and integrating multiple vendor technologies and products for Data Center Virtualization, Service Provider, and Cyber Security solutions.  You can contact Bryce and other architects at [email protected].

Bryce Trapier

Bryce Trapier

Technical Solutions Architect