Network Design: AI Data Center Design
- 16 hours ago
- 79 min read
Detailed post covering on-premises AI Focused Data Center Network Design
You have awakened one day where LLMs are taking over and your company management says it's spending too much on Claude tokens, they decide to deploy an open weight qwen model, therefore you are being tasked with your team to create a network for an Ai cluster that will be hosted on-premises in your data center. Sound familiar? Networking is now back in style and it's a great time to be a network engineer in the age of "artificial intelligence".
In the last design post we talked about active/active circuit design as a part 2 to my network edge series, but in this 6th installment of the network design scenario series we will take a detailed look at practical Ai data center network design. This isn't for someone pointing some API at ChatGPT and calling it a day, or spinning up an Ai foundry server, this is for those buying hardware and deploying something within their on-premises data center (DC). We will talk about essential DC elements and provide detailed related artificial intelligence topics.
The hyperscalers are all doing custom build outs of entire DCs with custom hardware, software, and protocols, so we aren't really targeting that for this post, more of a middle-market audience. We will cover high and low level design elements, different types of topologies such as small 2 server tuck-in setup, larger multi-switch spine/leaf builds and also larger DC considerations, looking at the practical design building blocks, like power, optics, and quality of service amongst other items. It's always nice to focus on a new greenfield setup and have everything perfect but in reality a lot of builds are integrating a brownfield so this comprehensive yet practical perspective will take that into consideration.
This post is for relatively advanced network engineering but as I have mentioned in other posts I try to also introduce topics and explain some basic items to those who are unfamiliar to strike a balance with readers. This post should be read in its entirety from start to finish to fully grasp the intention.
We will also exclusively focus on Ethernet networks using RDMA over Converged Ethernet (RoCE)v2 versus Infiniband (IB). I mean with IB you pretty much have to use the specific green team hardware stack (currently) with Nvidia unified fabric manager which handles everything for you. This is also more for someone building it out themselves versus ordering pre-built racks which is a common solution right now, thus if you're going that route please continue but know where this post is focused.
With Ethernet there are so many vendors and options it makes sense to focus on that, and it's what I have/would select in my designs. The content will include the items below to demystify the topic to give you ideas, education, and provide solutions.
Quick house keeping: I spell Ai with a lowercase I because I feel we have yet to reach true intelligence. Also just to be clear, I was doing bullet points before Ai, these are not LLM generated lol!
Basic Ai DC requirements:
Provide lossless network connectivity
Provide a 1:1 bandwidth subscription ratio to meet high bandwidth demand
Reliably connect the Ai xPU, storage and compute intra-cluster
Reliably connect the Ai compute to the wider networks
Optional: Provide segmentation for production and development (or multi-tenants)
Optional: Monitor the Ai network in more detail
Ai DC design challenges:
Minor packet loss results in massive efficiency loss for Ai workloads
PCIe 5.0 and beyond with kernel bypass means high bandwidth is needed
Physical space and environmental factors like power and cooling
Hardware procurement and pricing/budget
Business cases and ROI
Topics:
Foundational Ai Data Center Network Design Elements
High-Level Topology 1 - Two Switch Hybrid Design (~8-24 GPU)
High-Level Topology 2 - Small Spine/Leaf Hybrid Design (~160 GPU)
High-Level Topology 3 - Rail-Optimized Design (~256-512+ GPU)
High-Level Topology 4 - Monolithic Design (~512-1,000 GPU)
Key Low-level Ai Network Design Elements
Closing
Expected knowledge would be of Layer 3 routing (BGP/IGP), Vxlan with EVPN, multicast, spine/leaf Clos architecture, Multi-site, Ethernet basics, differences of IB vs Ethernet, optical basics, infrastructure cabling basics, rack placement, Layer 2 amongst other items. I also won't bore you with the same color-coded block diagrams you see in every Ai post!
This is for educational purposes only and I take no responsibility for anyone's network or their use of the information contained herein.
Foundational AI Data Center Network Design Elements
We always want to start with the business case when designing a network because that is what drives the direction for uses and future needs, along with winning the budget allocation battle! Although there might not be clear ROI yet for certain use cases of Ai outside of hardware sales, but that's not stopping adoption for organizations. One of the main motivators might be sovereignty and control of the data or apps which can help with employee productivity for specific business operations. This is why we have the on-prem focus here.
Most likely the builds here would be for inference servers and serving information rather than large distributed training of newly coded LLMs (ie. hyperscaler's training builds), it's more likely you are deploying an existing model that needs internal access for end users along with feeding it company data, so we can assume that is likely the case here; however we won't be discussing the application side in this post or the intricacies of like building the serving Ai etc.
Generally, we can assume there is a storage component and a compute component, so you will see some items related to both as we review topological examples. Inference typically needs less Ai compute which means there will be smaller builds out there for many organizations.
Price is another business case. As cloud compute costs especially for LLMs and xPU can add up quickly over time. A large cloud SaaS deployment and/or usage of external models plus licenses can cost over millions of dollars in a year which could easily justify an on-prem procurement. Take the case of something like the initial implementation cost with a 5 year depreciation of hardware could equate to 1-2 years of heavy outside 3rd party/cloud use, not even factoring in the increased performance and control that comes with physical ownership. It's funny when you read about current AI model uptime is sometimes under 98% whereas build-your-own you can likely get a lot better availability.
As you're undoubtedly aware there are some differences with Ai requirements versus standard data center requirements which we will outline in subsequent sections. Some people might still be deploying 3-tier distribution/core/access but the bandwidth density requirements alone really make that design obsolete for data centers. Clos spine/leaf offers predictable paths for flows, easy connectivity maps, consistent RTT, and repeatable "pods" for DC designs. I've also heard firsthand of people deploying Ai hosts onto a single-vlan flat office network which we aren't supporting and therefore will discuss some segmentation. We'll touch on multiple design options.
We aren't about to go into one of the dime a dozen Ai slop posts talking about things like traditional north/south and "new" east/west with Ai (even though that has been said for years). With that said though, it's clear there are some traffic differentiators with these networks that drive our design. For starters the closest thing to GPU-to-GPU (I will be using GPU and xPU interchangeably) flows is probably large storage transfers, these are called "elephant" flows for their long-lived high bandwidth nature. These are needed for the GPUs to exchange information between servers since there is likely a pool of GPUs amongst racks. In contrast to that, think of traffic such as a web app doing a small html or java data transfer to load a page, or some SQL communications between databases as more traditional traffic patterns. These can be characterized as "mice" flows since they are short and bursty. Still, GPUs can spin up to 400G line rate very quickly which isn't really done by anything else!
The problem becomes with standard converged DC networks is that the new xPU elephant flows can choke out mice flows; additionally, when the xPUs are performing actions like all-reduce communication if a packet is dropped before a job checkpoint it can drastically slow down the entire job being executed due to the fact all the xPUs need the same information. Thus the need to control throughput with things within an Ethernet build like explicit congestion notification (ECN) and priority flow control (PFC) come into play. This also influences our routing and hardware connective architecture by separating the traditional or existing data center to the new "Ai data center".
Hardware here will essentially be purpose built for this network because of the bandwidth and protocols needs. The ASIC and software will need to support RoCEv2 and will usually have 400Gb+ interfaces and 25+Terabit/s capability. These features are essentially standard on all new DC switches as of 2026 but it's highly likely your existing data center doesn't have interfaces of high speed at the necessary density, thus new hardware should be procured for this.
Figure 1 is showing our standard leaf/spine fabric inside a data center with a set of "service leafs" that provide outside WAN and security connectivity. This would be our base case topology. It's expected you already have a similar footprint to this, but if you do not, then consider the traditional items called out throughout the post.
A question that pops up a lot is how many leafs and spines? The answer is like a government contract: indefinite amount/indefinite quantity. There is no set amount for a network design; the right or wrong answer is the amount of ports you need now and possibly into the future which determines your leafs. The number of spines is determined by the bandwidth subscription ratio, future expansion, and amount of necessary redundancy. This is of course outside of any specific feature sets or software that's warranted.
Next in figure 2, we'll show that maybe we are adding Ai cluster servers to our data center, so we integrate another set of leafs and spines just for the xPU network back-end. This presents a hyper-converged setup where the storage is also within the servers themselves or is a part of the front end network. There will be more diagrams with different architectures later. The more performance you need will dictate if additional networks or "planes" are needed to expand to meet the demand.
We show BGP with VxLAN and EVPN overlay for the back-end, but as we discuss in the routing section we don't necessarily have a hard requirement for that in the Ai networks. We can support (and even prefer) native Layer 3 routing which is surprising to some people.
Bandwidth
One of the reasons to use the spine/leaf is that it is easy to calculate the bandwidth subscription ratio as there is always an equal number of uplinks from any one leaf switch to one spine switch. You've probably seen this before in the service provider world or in the old access closet with a 48 port switch and a 1Gb uplink (48:1 subscription ratio).
You'll need to be able to calculate this and ensure for every downlink you have an equivalent uplink of equal or greater bandwidth capacity to create a minimum 1:1 subscription ratio between downlinks and uplinks. This would indicate there is no bandwidth over-subscription. This is one of the reasons in figure 2 we needed to add another set of switches to create a mirrored topology, and that is because we need to do everything we can to meet one of our design challenges - to minimize packet loss.
The formula is very simple: you take the amount of downlinks and multiply it by the bandwidth speed, then you take the uplinks and multiply those by the bandwidth speed, then you divide those numbers.
4×400G = 1.6T and 2×800G = 1.6T therefore 1.6/1.6 = 1:1
With the rise of PCIe 5.0 and the GPU traffic bypassing the kernel thereby eliminating the CPU being put into the mix, we can see a closer to realistic theoretical max throughput hitting the network. Newer builds running PCIe 5.0 and using x16 lanes per GPU have a theoretical max about 500Gbps (~63GB/s) so you can see why 400/800Gb interfaces are becoming the standard with 1.6/3.2Tb just around the corner. We are also seeing native optics built directly onto the chip boards hit the market. Most high-end new designs are probably building with 800Gb downlink and 1.6Tb uplinks but we'll focus on 400/800Gb as that is more realistic for broader audiences.
Figure 3 shows a common Nvidia middle-market server, in this case a H200. You can see the middle compute OSFP cages which are there for the 8 GPUs within the system which are 400Gb each (we use a dual 800Gb optic for those). Then there is ConnectX cards on the left and right (v7, but v9 is out now) which we can select with a 2x 400Gb Ethernet configuration. By doing that we will structure one for the front-end and one for the storage network (more on that later). Though, it means that if we are moving to calculate bandwidth beyond the initial leaf switches you need an equal amount of bandwidth on the uplink. The 100Gb Ethernet is for the base command manager software to provision the unit and such, we won't discuss details on that.
Therefore for 2x H200 servers with 16 GPU x 400 Gb/s interfaces we need at least 6.4Tb/s of uplink bandwidth between both spines to make it a 1:1 subscription ratio! (ie. 3.2Tb/s per spine).
Of course that doesn't include the other 400Gb ports for the front end and storage which don't necessarily need a 1:1 ratio as they are for more traditional flows, however we still need to account for those. You can use all 4 of those ports for maximum performance and redundancy or just 2 ports with storage/inference traffic being mixed, the choice is yours.
There are a few ways to do this and unlike in figure 2 we aren't converging storage on the front end or within the servers and have storage nodes that have a dedicated network. Therefore, that means we are looking to have literally 3 different independent networks to support each of the use cases of GPU, storage, and front-end network as shown in figure 4. This diagram also assumes larger builds with elevated density for each, so multiple racks of Ai, traditional, and storage servers.
This is the high-level look at how a higher-end build could be created.
Although there is only 1 uplink per spine shown, it is clear we would need 4x 400Gb uplinks or 2x 800Gb uplink to each spine for absolute 1:1 ratio in the traditional sense for 16 GPUs across 4 leaf switches. It is technically correct to use equal-cost routing to spread out the load but with standard ECMP hashing this doesn't always balance things evenly as we will discuss in the load balancing section.
To continue, if we add 2 more servers for a total of 4, then we have 8x 400Gb down and 2x 800Gb up which means it's a 2:1 ratio. This is less ideal from a pure bandwidth perspective. If we go beyond that and have 8 servers which equates to 16x 400Gb ports per leaf in our example, then we'd need 6.4Tb of uplink. That means we'd need 8x 800Gb uplinks to achieve the highest bandwidth capacity ratio. You would spread your total number of uplinks across spines.
One thing you might want and what some operators have done is build a 1:2 ratio of downlinks to uplinks (so you have 2x the bandwidth required at the spine layer). This can provide future growth to easily add servers but also can account for link failures in the topology. For instance, if you are perfect 1:1 as soon as an optic fails you are in a degraded state. By having additional uplink bandwidth you take that problem into consideration. The caveat being the expense of added optics.
You can guess why needing 1.6Tb and 3.2Tb optics is becoming a necessity, think about next-gen with PCIe 6.0 needing even more bandwidth. There are a lot of possible combinations but I have an example of the 400/800Gb 1:2 ratio in figure 5 below.
Internally in most Ai servers like Nvidia's, the GPUs talk direct via the NVLink connections (others might use UALink or the PCI bus) so having only 1 server means there isn't a requirement for this calculation, but it's when there is multiple servers and GPUs being networked together where this comes into play. That's because of the way the data needs to synchronize among the workers. Using this high-speed interconnect within the server though allows for interesting communication flows and efficiencies as we'll discuss using NCCL and RCCL techniques to gain that. It is expected NVLink, parallelism, collective operations et al. has already been researched by you.
Quick side note. There's a debate raging in the community now whether the focus should be on the scale-up (eg. More focus on NV/UALinks) and creating denser single stack racks, or scale-out (eg. Focus on traditional build outs using spine/leaf) which means more racks that are less dense. I think we should look at scale-out here as that's the current industry deployment meta for DCs. It's not that scale-up won't continue, as Moore's law is a thing of course and the future photonic fabrics will facilitate this with better efficiency, but with talks of like a 1 Megawatt direct current powered single rack being tossed around, I kinda throw a inquisitive look of comedy at the thought of that.
Anyway, if you are running many scalable pods (beyond 128 leafs) that need to connect everything, it is common to run a "super spine" layer which at certain scales could warrant creating a 3-tier Clos with the 3rd tier connecting each of the pods. When it comes to this network topology obviously to get a perfect 1:1 bandwidth ratio would be very expensive which means here over subscription at the top layer is acceptable to a certain extent. There has even been testing where a large 2 tier Clos can be acceptable for multi-pod because to a lesser extent most traffic will be intra-pod versus inter-pod.
Finally, once we achieve the optimal bandwidth ratio there are some more mechanisms like Priority flow control with explicit congestion notifications which require special QoS marking and classifications and are setup on all of our network ports (specially on the spine links). We also need to consider our tunneling for QoS if present. We will cover this in the low level design section.
Environmental
I briefly want to touch on the environmental aspects which are often overlooked in network design. At the big shops there is often dedicated teams for this, but for smaller builds it's often the network designer who needs to take these things into account.
You've no doubt heard a lot about the power generation shortfall for DCs when it comes to Ai because of the insatiable need the GPUs have. Consider the previous example of the H200 server, these units have 6 power supplies that are 3,200 watts and 16 amps each and the newer B300 has 12 of these. Moreover each of these servers needs a minimum amount of power supplies online to stay operational. Think about how traditional servers or network switches might only have 2 power supplies, but with these GPU servers having 3-6x that, this causes more heat generation which thereby needs more cooling in addition to that. Some vendors require the minimum number of power units to be online while others do not require a set amount and determine it based on the power consumption of the accelerators. Verify with your hardware vendor of choice.
Considering the average cabinet traditionally only as like 10-15kW of power delivered (and heat dissipation planned) there is a greater requirement for these setups. We're not going into deep power calculations but simple checking of max draw on the power supplies then referencing the max wattage and amperage supported of the receiving power distribution and its outlets is recommended at least. This also illustrates why you are hearing about liquid cooling in a lot more for these projects.
Providers are deploying 60+ kW per cabinet now, and sometimes this might not be available in your current colocation cabinet or cage, so be prepared if you have to deploy into a new building or campus to accommodate the power and cooling for a LLM deployment which could add to the project expenses.
For power circuit delivery it's possible you will have 4 instead of the standard 2 delivered to the cabinet thereby needing up to 4x power distribution units (PDUs) to plug everything in, but it depends on the data center's modus operandi and electricity availability of course. But think about a simple single phase standard rack PDU might have 1x30amp breaker and 2x15amp banks per circuit, this is not enough as we illustrated above.
Count that each of the power supplies has a max draw of 16amps, so we already need at least 1x 20amp circuit per power supply, and in the case of the B300 it needs a minimum of 6 of the 12 power supplies to operate minimally, the H200 needs 4 of the 6, therefore we can deduce that there is probably a max of 2-4 of these servers per rack depending on the power availability.
I know horizontal PDUs are being used more often for Ai racks with 3 recommended but this could be difficult to replicate due to the supply chain shortages. Potentially you will have to use standard vertical PDUs and maybe automatic transfer switches (ATS) in order to ensure that even if a circuit goes down or a PDU has issues the power supplies can stay online. There are some beefy PDUs pushing 3 phase 100amps with 28 kW power capabilities on 208v. Also think about needing 2-4 of the ATS's depending on how many GPU servers are needed, plus you need to power all the switches if you are running top of rack. You might be surprised once you open a PDU and see one of these large connectors, so be sure to cross-reference your inputs.
For the switches and storage sometimes it will be a dedicated rack that's middle-row or end-of-row. All those devices are housed so that could be a separate cabinet from the servers and powered with lower end equipment versus higher end PDU/ATS.
Hopefully this got you thinking about things in a new light, as for an example if you had 6 power supplies, and the minimum needed is 4 you would conceivably use an ATS, but unfortunately the ATS that are available (hypothetically) only have 20 amp circuits, so you'd need 2x ATS or 1 for each of the redundant power supplies. Then the other 4 would be plugged into a bank on either PDU. So power supplies 1 and 2 on PDU left, 3 and 4 on PDU right, then 5 and 6 each into individual ATS which are then plugged into individual banks on each PDU. This was illustrated above in figure 6.
Figure 7 below shows a sample from a Nvidia guide, check the reference link for more info and examples for connecting their GPU hosts.
You might elect to use ATS's for each power supply but of course this increases costs (and rack space) and there is a supply chain shortage like I alluded to, plus I've heard lead times are insane for the most sought after models, therefore you have to balance things in your design. There could be a bigger limitation on the power and cooling capabilities in your deployment versus something like bandwidth and switching. The power considerations for your LLM build might take more time for you to plan than expected. This subject goes very deep and should not be overlooked, I've seen discussions of how optics choices can eat much of the power budget, and as we talked about in the bandwidth section the amount of optics can add up. Be sure to research your power plan in depth.
Storage
I just wanted to touch on storage briefly as it's a key component to the build which like power, is sometimes the responsibility of the actual network or systems teams deploying the equipment, as opposed to a dedicated storage team.
Like we showcased in the bandwidth section, speed is a key factor in the design decision process along with the purchase of the build of materials. Therefore it is important to consider the PCIe bus speed of the board and storage you are planning to use. Spin disk is likely only plausible for long-term storage (as is the standard), but I know some of you might try and get away with it for production work loads. NVMe storage supporting PCIe 5.0 or even 6.0 with high I/O capabilities now is a must for high speed access to your Ai information. Another important aspect is the amount of storage as with whatever you plan on training or needing to inference, the amount needed scales linearly with text versus pictures versus video (which also is a part of the 'tokenomics' calculations).
Lead times are very high on this type of equipment in the last 12 months along with price increases across the board; so unless you're a big org with a lot of pricing power good luck with the latest and greatest but I probably wouldn't settle for like a PCIe 4.0 system long term regardless of availability.
Speed is the bottleneck as is the ongoing theme here. Most storage as far as I know is only running x4 lanes so obviously it's not as fast as the x16 xPU but still around 16GB/s theoretical. Pure storage, Netapp and similar solutions all provide solid options if you are looking to deploy a dedicated software + storage bundle for this system as is the common case. Be sure to fully understand the BOM as sometimes they require to use or just bundle by default their own "fabric modules" which are essentially switches to use with the storage which you might or might not desire. Back when FCoE was big this was essential as special chipsets were needed to switch it at high speeds but most likely you aren't going to be using fiberchannel.
The storage protocol is important, since most of this is linux, NFS will be common and I recommend using NFSv4 as it has a lot more security builtin like authentication with kerberos or LDAP which can integrate into your AD environment along with encryption in transit which v3 lacks. If stuck with v3 then IP-based allow/block listing will be important since the authentication mechanisms aren't as robust as version 4. RDMA is also possible for storage access but your mileage may vary, it could be good for the GPU servers to utilize this to access the storage whereas the NFS will be for other related VMs. You also might have NVM-oF (over Fabric) for a very high speed transport protocol.
In some setups like hyper-converged servers there is often some room for a few SSDs onboard so there is the possibility for like POCs or dev/test that you don't need a ton of storage up front, additionally a lot of the modeling is done in RAM since that is the fastest way to access data. However it's important for data storage or information for training that a dedicated repository is provided. Some might just pull their updated models off the internet when they want to redeploy or update images but having this on local storage with 400Gb interfaces between devices offers far more speed and security than accessing and downloading data over the WAN.
Back to security, a dedicated VLAN/network will be a good thing to have for the artificial intelligence servers and storage shares to be accessed through, this segmentation can allow direct access over high speed switching with both the server and storage appliances having interfaces on that dedicated subnet/VLAN. This is versus traveling via firewall which can be a bottleneck with some slower speed vendors, but it would be assumed the initial access to get data onto the shares would be through a firewall with inspection (along with on-board storage appliance inspection). We'll revisit this in the segmentation section.
If your existing DC is HCI already then it can be debatable to deploy a new dedicated storage network with multiple switches, outside of the storage fabric switches, to support dedicated repositories for this use case. With a few special servers and storage appliances you can undoubtedly get away with having them on your dedicated leafs for this project such as in the topology examples in this post. Otherwise due to scalability and bandwidth needs it will be good to budget and provide separate dedicated fabrics for each use case (front end, back end GPU/xPU, and storage).
Optics and Cabling
If you've been following the industry closely as of late, along with microcap chip stocks, you've probably seen all the hype and endless discussions about the power consumption, the rise of co-packaged optics, and the millions of different options now. It's clear though that the density of these LLM builds as we showcased in the bandwidth section of needing at least 10-12 interfaces per server along with leaf to spine needs, the amount of optics can balloon your budget.
With something like 100k optics for a large design being easily possible and considering different MTBF rates even at a failure rate <1% you're seeing multiple possible optical failures per day at that scale! Plus each optic needing more than a few watts of power each, multiplying that by hundreds or tens of thousands starts to add to the power budget.
Nevertheless, if you aren't up to speed then here's a few new things to get you updated. First is that standard LR and SR singlemode or multimode SFP are not as simple now because of the multitude of different choices based on the optical bandwidth lanes that are needed for the higher speeds, along with different form factors depending on the OEM. This means details like compatibility are very important in optics selection before ordering. Secondly we can celebrate as the need to roll fiber connectors is not as widespread now due to the new MPO/MTP connectors that bundle multiple fiber strands into one cable, as opposed to the old SC/LC that is 1 fiber pair. We are still using LC connectors in certain areas but for many of these 400Gb and above if you aren't using a direct attach cable (DAC) then the connector will likely be a MPO type. See Figure 8a and 8b.
Notice there are 8 LC pairs on the breakout cable, this is because there is 16 fiber strands in that MPO cable. They come in 8,12,16, or 32 strands, but 12 or 16 is what you will probably be using. Moreover notice the 16 holes in the connector for Fig. 8a representing each fiber strand.
These are much easier to plug in now and we don't need to worry about the polarity as much, though there are still tools that test to confirm before hand. Side note, the MTP connector type is essentially an MPO but it's from a specific manufacturer that patented it and as far as I know is superior due to less loss and tighter specifications which makes it the better choice for patch cables.
Early on in my research for a project it was very clear some vendors were behind the curve as there were only a few 800Gb optics on the market and most big name vendors didn't even have 800Gb ports available on their switches. Fast forward and vendors are even using their competitor's silicon for their own switches to keep up. It was difficult to find a Mellanox OSFP form factor optic that matched with equivalent Cisco optic for example, however there are a lot of aftermarket optics which are good and work great with the server NICs.
No I'm not mistyping OSPF and yes this is the practical guide, so we aren't about to start talking about the benefits of PAM4 modulation and all that. Though we do need to make sure that the form factor of your desired optic matches the switch or server you are using and that the chosen variant of the modulation/lanes matches and is compatible.
There's some compatibility with QSFP style cages like QSFP28 might fit in QSFP56 or QSFP112, or a QSFP112 might fit a QSFP-DD port, but QSFP-DD won't fit in any other cages and so forth. OSFP and QSFP are not inter-operable from a port cage perspective. Likewise SFP+ or SFP28 obviously won't fit any QSFP port, thus if you still have devices that are using the smaller SFP form factor and your new switches are QSFP only, then breakout cables are what you will need.
For our focus on 400/800Gb, DR4/DR8 is likely going to be the best price/performance/availability with QSFP-DD being common for non-green team switches and OSFP for the 800Gb switches/servers. OSFP outside of Mellanox was hard to come by about a year ago but it's at the forefront now with all vendors' new models supporting from what I've seen. There are multiple competing standards right now regarding modulation and form factor, so we'll need to pay attention on who wins out long term.
Figure 9 below shows all the different types of optical form factors and modulations. One key thing I want to highlight is the FR/LR versus the SR/DR. The FR is a CWDM type which utilizes standard LC dual fiber connectors (1 pair) and the latter uses MPO-12 or 16 cables (6 or 8 pairs). These optics are not compatible but have different use cases. Moreover, since the FR/LR uses wave multiplexing over a single fiber it cannot be broken out like the SR/DR.
If you picked up on the MPO fiber density you might be wondering what about longer runs like cross-connects or inter-DC or cage to cage etc. and yes it requires far more cable infrastructure to support. Thus comes in the CWDM options like FR4 that can utilize the existing fiber infrastructure and require LC connectors indicating they are great for Inter-DC building or cross-connects. This would be like in the instance of having to deploy to a scale-across to a new building because of power requirements of your Ai rack. The FR4 and LR4 have decent distance so even for intra-city this is the logical choice for 400Gb over a dark fiber right now. The DR4 and DR8s is what you are going to be using on short runs inter-rack or inter-cage that exceed direct-attach lengths.
DAC passive copper cables are also making a big come back due to their price to performance, but you are only going to be getting 5-7 meters from these. They are found in 400 and 800Gb and I'm wondering if 1.6T can be stretched for these in the future. Definitely weigh the pro's and con's during your decision making process and research, although there is a lot it's easy to digest.
For a build with say 3 racks in a new building away from your existing network, 2 for Ai servers and 1 for network/storage, I would elect for longer DACs between cages and then a CWDM for a longer CC back to the existing network. If there was more racks spanning longer than 5 meters you need to make the decision if you want to run DAC and/or AOC and/or traditional optics with jumpers. The caveat being the complications of having different types of cables and optics from an administrative and supply chain perspective. Although it will be more expensive to only have traditional optics and jumpers, your administrative overhead would be less. If running many racks and very dense cabling think about the weight and support and air flow restrictions large bundles bring. These are things you need to decide on before implementation. Counteracting that though is the power consumption of each which begins to add up at scale.
To close out this section the forward error correction (FEC) scheme is another thing that can be a conflicting item and something to read up on for your switches and server NICs. Let's say everything is congruent and you did all your due diligence but your interfaces still are not getting link; trying speed/duplex setting hard coding, flipping polarity (if LC) and all that but you still can't get connectivity. It could be a FEC issue needing a port config change. Usually things are auto by default, except if that isn't working then hard coding the various schemes like cl91 rs-fec might correct the problem. Additionally, I would say this might be more if you are doing breakouts to like 400->100Gb or 100Gb->25Gb. It's weirdly funny when you're doing an implementation and you researched and found this could be something that pops up for a certain appliance and then it does manifest by no link between your interfaces, it's an issue but expected and you can fix it!
This subject, like the others here, is worthy of an entire post and really just scratched the surface on what is needed, but I wanted to mention to keep an eye out for co-packaged optics (CPO) or Linear-pluggable optics (LPO) which are essentially optical transceivers built onto the chipset/ASIC. These are going to be more widely deployed for larger build outs I think and saves on cost and power while reducing latency, however it sacrifices flexibility and serviceability. Another thing to add to your reading list.
IP Routing
So far what we've discussed has been straightforward but now we must talk about a controversial topic - IP routing. How are you going to route everything? There's many options and I will showcase a few to help influence the decision if you are stuck. Even though we are showing different designs it's expected you're pretty researched in this area and don't need step by step configs (as there are tons of guides on this already). We'll look at some standard data center or front end for Ai and then look at some Ai pod routing. I'll display some config examples in each diagram to better showcase what it's about.
As with any DC network we must route in and out of the server farm in order to access services, and we do this reliably of course, and we mentioned that in the requirements: Reliably connect the Ai xPU, storage and compute intra-cluster & Reliably connect the Ai compute to the wider network. However we know that with the xPU network we have to take to a whole different level, which is really achieved via quality of service and bandwidth or scheduling but the routing is really what gets the job done.
A few questions you will want to ask up front or maybe you already imagined this based on your existing brownfield network, but that is are going to run the routing protocol from the hosts, layer 3 to the hosts, or run traditional layer 2 to the hosts? Another important question is if this is going to be a multi-tenant service provider type network needing full isolation for different customers or not and if you are running MPLS already in your network (or maybe want to).
Are you IPv6 throughout or maybe considering using it here? Should this mirror existing DCs or be different and standalone? How many VMs/switches and even locations could this grow into? Your choices now could mean headaches or sleep well at night now and into the future. Only you can answer this exclusive to your org and business cases.
I recall recently (and many times before) of people asking on reddit r/networking about if they need BGP/EVPN for their data center, and the answer is always it depends. Many will say you have to, but on the contrary it's perfectly fine for some environments to simply use a VxLAN layer 2 overlay with multicast control plane over OSPF or IS-IS. The first item to think about is knowledge and capability of your team then the size and future scale of the colocation foot print. Is segmentation needed but you are a single organization? Do you need multi-DC fail over to cold or hot sites?
If you are running a single or maybe 2 DCs with like a couple hundred VMs it is probably perfectly fine for standalone Vxlan and OSPF and is arguably better than just layer 2 since there will be blocking with spanning tree. We always want a non-blocking topology in the data center.
A nice to have is layer 3 migration capabilities (like with Proxmox or Nutanix) if you need multi-site fail over, so you don't have to extend the VxLAN domains between sites. Therefore, you can run small multi-sites without extending layer 2 and without EVPN, but more sites and bigger scale is where EVPN starts to make more sense if the additional complexity justifies the need for that service level. With a layer 3 underlay we can get ECMP to better utilize all uplinks and accordingly better meet our bandwidth requirements. This configuration means we are going to run layer 2 down to the hosts from the access switches which is probably the most widely understood host connectivity method. This idea goes for a small Ai network like in HLD 1 or 2 where we can run some basic L2 with VxLAN if desired.
Figure 10 covers this initial standalone DC routing. We will keep with the 2 spine and 2 leaf 'pod' example to keep things consistent in this section. The idea with any of these is to make a decision and keep a repeatable uniform building block as we expand the network. The spines do not have any Vxlan information only IGP and multicast. They are anycast rendezvous points for PIM-SM in the overlay network's control plane to pass broadcast, unknown unicast, and multicast information. Notice I have some specific Cisco configurations shown for that along with items like MTU which is needed for the Vxlan header overhead. I also list VxLAN specific information.
Next is the tried and true, a recommended routing choice for most DC environments, and that is the BGP EVPN with Vxlan, without using multicast control plane. We swap multicast for EVPN for things like VTEP discovery and mac learning for layer 2 but like everything here you still have the option of running PIM-SM if you want! To continue, this way is more efficient from BUM traffic handling as well since it avoids unnecessary duplication. If you are running multi-DC with anycast IPs this allows for seamless fail over from a layer 2 perspective between sites and is the current best practice from an industry perspective. Likewise to the previous option I also consider this for back end layer 2 deployments if not wanting to just run pure layer 3 for xPU and want more flexibility and options in the future.
On the BGP side you can do an eBGP model with each of the leaves having different ASNs and have the spines with the same ASN, however if VRFs are needed as is the case of the multi-tenant front-end isolation scenario, then you could have issues with route targets being auto-generated if you're using auto-RT type config. This is due to the ASN being different and thus causing problems. Therefore it's plausibly better to just recommend the iBGP model where the spines (maybe not all) act as route reflectors and all switches have the same ASN for scale-out. Although, for the back-end builds here like with LLMs which are mostly isolated and won't be using VRFs, as you'll find out, this decision has less implications for the future since it's standalone most of the time. Thus, the eBGP option is the simplistic scale ready choice; or even just straight IGP can be a great option for small less-technical teams (I know a few will ask that). As you can see there are many choices to reach consensus on.
Still, there is also another BGP way to build with all spines with an ASN and all leafs with a different ASN if you want to play with BGP next-hop features. In any event, despite us planning to use dynamic load-balancing which we cover later, we need to ensure you have set your BGP max paths to more than 1 and to a number equal to the amount of uplinks/spines you'll have for proper ECMP. This also applies to your IGP if running a high number of spines as most vendors have a high equal-cost path count by default for IGPs.
In the next image which is figure 11 I have an Arista example of an iBGP EVPN Vxlan type structure. I again show OSPF but you could use IS-IS instead of OSPF if preferred. In addition, I haven't generally advocated for anycast gateways in the Clos in an EVPN network (as a lot of guides show) as in my opinion that is only for a multi-tenant MSP/CSP and should really only be used there. Here at networkdefenseblog we push for secure centralized gateways on a firewall that is appropriately sized for your traffic volume. There is a setup we will check shortly that shows when we can use gateways on the leafs though, but that's for the back-end.
Speaking of centralized firewalls, how would this look based on these designs? By centralizing the subnet gateways on a firewall we can ensure we get full visibility for in/out routing of the networks and have easier automation and control of firewall rules. We want to prevent our agents from escaping of course. Managing ACLs or custom vendor SGT type segmentation can be very cumbersome even with their designated software, whereas with firewalls there is plentiful automation tools and management software choices to make life easier. I would say to have a zone per-vlan, which is then tied to a VNI and layer 2 overlay tunnel which is isolated between the firewall and the server/VMs etc. unless you're running L2 VPNs with SR, we'll cover that after figure 12 which shows the centralized firewalls hanging off the pair of service leafs. Then you can summarize routes on the firewall to the wider network for route table optimization.
On the topic of service leaves I usually lean towards the MLAG/VPC/MC-LAG type setup to have the switches appear as one. This allows for port-channels and simplified the routing in my opinion. You can run firewalls, routers, specialty servers or other appliances that don't fit the standard leaf/host model. With EVPN remind yourself to get the ESI configured for the multi-homing aspect with the firewalls or other layer 2 devices that might hanging off of the pair if applicable, so the type 1 & 4 routes are advertised properly in BGP.
As for the DCI that could be from the service leaves or from dedicated switches or routers and probably using eBGP to connect the islands or pods. For an addition to an existing fabric that already has firewalls and such I would just DCI between the border leaves or if it's a few switches then have the new Ai network leaf switches connect to the other network's spines and integrate directly into the Clos as we do in HLD 1.
If you're looking for business cases for IPv6 outside of the usual no NAT, more addresses, better performance, then there might be some additional incentives when looking at your new DC routing design. For starters you can run BGP unnumbered which uses the v6 link-local addresses for neighbors so you don't need to worry about p2p underlay interface IPs which can be good when running thousands of ports. Plus you could technically run BGP as both the underlay with ipv4 unicast AFI and overlay EVPN AFI for simplification (simplified depending on who you are). One thing though is recall if you are running OSPFv2 with IPv4 only and looking at doing ipv6, then that means you'd need to run OSPFv3 to roll it out, this is why IS-IS can be a better option to future proof your greenfield implementation.
For the back-end you are able to route RDMA Infiniband traffic over Ethernet/IP since it is encapsulated in UDP. I've shown the long-established layer 2 overlay way of inter-leaf above but in a highly scaled up back-end network for xPUs as mentioned you could run BGP or an IGP only from your leafs/spines/pods. Then you'll run p2p L3 links to each xPU and then advertise the /32s or /128s of the xPUs into the network (so you can straight layer 3 route from the host into the fabric and between xPUs). Alternatively, for a rail-optimized design like in HLD 3, you could conceivably have each leaf be a /24 or /64 and inject that route into the table while having each xPU use that leaf as its default gateway, this would allow you to still run L2 to the host but with native L3 only on the fabric. The later is my currently preferred option.
Know that these new app communication libraries on the hosts of NCCL and RCCL will discover the topology and know which GPU connects to which leaf and such; thus intra-cluster and pod traffic will be largely self contained using the UA or NV link. That means for smaller builds with Vxlan and layer 2 for the compute, we might get away without a default route. An example would be if on server 1, GPU 1 on switch 1 needed to talk to server 2 GPU 3 on switch 3 it would use it's internal server link to send the traffic to GPU 3 on its own server and then send it over the switch 3 leaf to server 2's GPU 3. Consequently that traffic in the same pod will be self-contained so you wouldn't really need a default route on the server. This is why we said in the bandwidth section although there is a need to have the 1:1 ratio, there are exceptions and circumstances where we can optimize with less optics to balance cost with the true performance requirements.
However, if GPU 1 on switch 1 in pod 1 needs to communicate with GPU 5 on switch 5 in pod 2 then you will need a default route on the host. Although I hate policy based routing and never condone its use, here it does make sense from the host perspective to accomplish what we need. A few vendors are showing that in their build guides right now. We'll use individual route tables with PBR (nmcli, netplan depending on flavor) to give each GPU a default route if it needs to communicate outside the pod. This is really the only time I'd advocate for its use.
Check out the next diagram where we take a look at how this preferred option would look for AMD and Nvidia. We show both v4 and v6 addressing here with a possible start of an IP scheme, however this depends how many leaves you select and your overall design if you are going to connect it to the outside world or not etc. Consider that when determining future scalability.
Continuing, Recently I've seen some design preferring SRv6 for Ai designs so that possibly opens up some simplification depending on who you are. I think this nanog presentation shows a good example from Azure of that use case but that is very custom and future-thinking as they are using static routes with automation. That does mirror other things I've heard of about the static pinning of SRv6 routes which requires a controller though. There is compatibility between the SRv6 SIDs and things like Kubernetes and Linux which means containers or hosts can participate directly in the segment routing infrastructure which provides a scenario to simplify your stack if you want end-to-end layer 3.
Nevertheless, doing SRv6 with an IGP its still an option if you are looking at running segment routing in your environment for a greenfield build. However SR-MPLS with layer 2 VPNs is another option you can utilize in the data center instead of VxLAN if you didn't want to introduce another protocol and need segmentation, and if you're already running segment routing. I would probably advocate for SR-MPLS versus SRv6 due to various reasons like compatibility and adoption but I'm sure depending you you talk to one will argue for one or the other and SRv6 designs are floating around recently for Ai so...
Some of the reasons you might want to do SR-MPLS is that the IP header overhead is less versus VxLAN even with multiple SIDs stacked up, which implies slightly better performance along with integration into the wider network or other locations for DCI (again if running it already). This is especially true if there is different customers and VRFs that are geographically dispersed across multiple campus buildings and you need to extend their fabrics.
Also with TI-LFA the failover can be sub-50ms which would means you can get fast fail over and rerouting without BFD or super aggressive IGP timers. That means things can route better in the case of an optic or DAC loss on the xPU back-end network which requires minimal packet loss. Speaking of that you will want to run aggressive routing protocol hellos in order to account for link-failure reconvergence scenarios.
Finally, back to the multi-tenant or need for VRFs. Just remember that friends don't let friends use VRF-Lite in the DC or WAN for that matter. Please use Vxlan overlays or SR with L2 VPNs or just layer 3 from your hosts if supported, save us all from that architecture. For the brownfield network of say running IS-IS and SR-MPLS as a core service, but perhaps the business wants to "re-brand" as Ai or add new services, and therefore needs a few DCs stood up, you can still map MPLS labels between VxLAN sites with EVPN/SR integrations.
We talk a little bit more about how the overall routed and IP architecture looks in the network segmentation section if you still are left with some questions.
MRC protocol is something that's recently been circulating and will be another topics to add to the reading list which combines a lot of existing mechanisms for a holistic optimization of traffic routing for larger networks.
Next we'll wrap up this section with monitoring and then proceed onto the High-level topology options.
Monitoring
When it comes to monitoring your LLM serving network there are a few key items we need to review. First is that standard SNMP monitoring will probably not catch the necessary metrics in a heavily loaded Ai DC network.
For example a traffic microburst that exceeds the port's buffers for a few seconds and causes packet loss will not be seen if you are using standard 5min polling metrics, even at 1 minute it is not likely to show up in a graph, but if you try sub-1min polling your CPU usage will increase on the switch and might not be supported by the software either. Likewise with reporting using 95th percentile as is typical won't catch these, thus you still need to check 99th percentile reporting.
Streaming telemetry is the go-to here likely with gNMIc or similar tool. This will allow you to graph at granular levels to see something like the quick microburst and possibly integrate tools to alarm on that. If your tool doesn't support that or you are not licensed (certain vendors lock that behind a license so think about that for budgeting) then at least use a 1 minute polling for your port metrics. Some vendor management software come with streaming telemetry as well but if not you will need to go custom with a collector and grafana. I've even heard that 1Gb is not enough for some streaming telemetry systems as well, so consider that before OOB switch procurement and the additional bandwidth if OOB is an independent network.
You'll want to monitor things like the buffer size (growing buffer means there is back pressure) and queued depth to determine if there is congestion happening, also things like ECN and PFC metrics to see if these are triggering excessively and where. We cover these in a later section.
Even though we might be running centralized gateways on firewalls to get flow logs there, it still is wise to run netflow/sflow etc. to check on QoS markers within the fabric and to get your flow visibility in another format for reporting and troubleshooting. You can take this a step further with a custom or off-the-shelf solution to capture packets related to RoCEv2 headers to record packet Qpairs with CNP generated, congestion points, and/or notification points etc. for when packets drop to help identify where it is happening.
There's also certain hardware show commands which you might want to have an automation periodically run to capture for historical needs or for alarming, for example on a Nexus switch the "show queuing interface <int>" can show a lot of details about dropped packets and queue depth.
You'll want to run iperf3 tests if possible before production workloads are sent over the network to validate your QoS and optics/cabling integrity. You wouldn't want things to go under load and then start to see errors on the go-live date, therefore it's wise to run synthetic qualification tests prior to production to test and validate your monitoring, alerting, and hardware integrity. There are usually benchmarks with NCCL and RCCL built-in. Even better would be to partner with the development team and have them stress test the servers with their preferred model for a sustained period and network before production go-live. Later when things actually matter you'll want to preemptively be able to identify where the problems might be before something goes wrong with a job that's running, and if those jobs don't have well defined and supported check points much time can be wasted from work load stoppages.
Power is something that should also be monitored at least from an alerting perspective. During the PDU/ATS process if something was not cabled incorrectly like too many power supplies under 1 breaker then once your servers start to run higher workloads which need more power for a xPU you could see power threshold alerts. If that ends up happening do a re-review of all the outlets and perhaps tune the threshold as well. For instance if the warning alert on a PDU is at 13 amps, but you're seeing a 20amp bank hitting about 13.9 amps under load, adjust that warning threshold to 14.1 maybe. It's assumed here the overall cage power and cooling is monitored by the facility you are located in.
Likewise as the scale and amount of optics increases you will see more failures so being able to pull metrics like power draw, loss, temperature and such to possibly preemptively identify degraded optics is vital and more importantly on what switch port for replacement!
Be sure to develop a strategy for the new DC you're creating before procurement or as you're waiting for hardware to arrive as some of the standard monitoring will not suffice and could warrant new tooling to get better visibility.
High-Level Topology 1 - Two Switch Hybrid Design
For this first network setup we are looking at more of a tuck-in implementation that will apply to a lot of people due to the fact they will be deploying a few servers for a new cluster to accommodate the minimum requirements into their existing environment. I developed a small slight "hybrid" of the traditional DC access and the rail-optimized design used here and for the HLD topology number 2. We have the GPUs split between switches as opposed to all ports from each server connected to a single switch individually. The following example is probably for around 8-24 GPUs.
This can handle two servers like DGX or equivalent for a small deployment where cost is restrictive and/or a small foot print is desired, where some rail-optimizations are gained but without the entire build out needed. This should cover production and development use cases which is one of the minimum requirements we need. Plausibly a lot of budgets might be constrained this year because of the elevated cost of everything tech and Ai, thus a smaller POC type setup is what you could look at first.
One of the newer 64 port 800Gb switches is fairly attractive for this structure but those can be hard to come by; so a more common 24/32/36 port switch is likely. 2x 64 port full 800Gb 51.2T switches could definitely handle decent expansion with this footprint. Something like 4 servers each with 8x GPU would be 16x 400Gb ports per switch, then at least 2 more 400Gb ports for front-end/storage, then with the inter-switch links and spine uplinks we're looking at ~60-70% of the switch ports used. There's also out of band management (OOB) which we aren't diving into but generally that will be a typical 1/10Gb SFP/SFP+ or copper type switch to connect to those specific management ports on the switches and servers.
A few options for any of these plans in this post might be the Cisco 9300-GX2 type switches, Arista 7280R4 series, or Nvidia SN5400 series. Chassis switches were more popular early on in the evolution and still viable for larger build outs, especially for spines or middle-of-row switches as we will look at later but that is cost prohibitive for this topology.
Figure 15 takes a look at the high level of what this would look like. We have 2 switches and place them in a MLAG/VPC or similar to make them appear as one and we cable our two compute nodes and connect them to it. The 2 switches are then connected to the existing spines in order to connect to the virtual host and storage infrastructure in the other building, along with the WAN for north/south traffic.
We assume here that the existing spines are older and do not support full density 400/800Gb so we use 100Gb QSFP28 LR optics (recall form-factor compatibility in the Optics section). Presently it's commonplace for high power racks to be in different or new buildings, thus we make that the case here and have 2 separate cross connect fibers from our hosting provider to our current leaf/spine rack to facilitate the spine connections. Recall from the optic section we prefer the LR CWDM type lasers for cross connects if we are doing >400Gb.
Since these are cross connects and our LLM/Agents will be having sensitive corporate data, MACsec is recommended for that scenario. Double check your switch's specs but a lot of these new chipsets are full line rate MACsec.
Due to the 100Gb limitation, and the fact there's only two switches for this setup, we have multiple 400Gb inter-link specifically for the GPU-to-GPU connections via an isolated VLAN only on those two switches which would have dynamic load-balancing; all other VLANs/networks participate in the EVPN/VxLAN routing environment to connect to the centralized gateways for the front end and other virtual machine's communication in the existing DC.
Like we mention later in this post about setting up K8s front-end for your inference VMs which could be on your current hardware and not on the new hosts, the actual hardware xPU power will need to be deployed in the DC and should manifest as new hardware. Therefore we need to grow the network to accommodate this new system. It could be repeatedly scaled but is not necessarily intended to and likely need spines to scale out laterally, but could be replicated for a geographic individually diversified roll-out. This is intended as a limited brownfield setup for smaller scale.
Below on the next diagram we show how it would look like in more depth. As with this and the next topologies, some of the config details were covered in previous sections and some will be in the low-level design section. Note the optical choices in the call outs. Also the split connections to illustrate the 800Gb to dual 400Gb breakout split for the GPU interfaces.
Total port count per switch:
4x cross connects for spines at 100Gb
8x GPU ports at 400Gb (4 from each host)
2x front end and storage at 400Gb / VPC Link(use more for performance/redundancy)
10x inter-switch links at 400Gb (would be less with 800Gb support)
1x Provisioning port (specific use case for DGX)
1x Switch management or keep-alive link between them
= 26 QSFP ports out of 32 total
You can see why a 64 port would be better as it could support 4 servers in this type of configuration which fits a common sizing for SMB/Mid-Market as that alone could be a million dollar plus capex purchase. We still have a few ports remaining which could leave room for a dedicated storage appliance if warranted or using both ports on the NICs for better redundancy.
Notice finally we have DAC cables inter-switch since the vendor is the same and is supported to save on cost for the short distance intra-rack, whereas between different models and vendors we use standard optics from each of them for better compatibility. There's a few different schools of thought there and that is my preferred methodology.
Next for example 2 we will expand to a larger blueprint and then for number 3 we will move to the super pod rail-optimized type structure.
High-Level Topology 2 - Small Leaf/Spine Hybrid Design
Building from topology 1 where we met the absolute minimum requirements for a multi-server cluster, we are expanding that here where budget might be healthier and are looking to deploy more servers. Be that as it may that the rail-optimized design is the preferred for Ai data centers, this HLD is intended for any build that is less than 8 leaf switches because with the parallelism xPUs use, most or all solutions on the market have 8 cards in them. SMCI x14 series (eg.Gaudi 3), Nvidia A/H/B series, AMD MI300/350x etc. all have 8 cards for various reasons, therefore 8 switches to key all 8 of the individual GPUs together between each specific one on the servers is ideal (eg. match gpu 0 on each host to a specific switch, gpu 1 to each host to a specific switch and so on). We will cover that 8 switch layout in the next section HLD 3.
It's expected that the networking in the budget for your build while important, will not be as prioritized as much the compute, therefore we need to plan for a scenario of more than 2-4 servers, but less than ~11, we will use 10 for this example. In light of that we are planning for tradeoffs when we look again at having 32 port switches for the leafs but with 64 port for the spines, they are all 800Gb capable.
First off we have 4x 32 port leafs serving the GPU communication, and 2x 32 port switches as service leafs for the front end, storage, and DCI, this is along with 2x 64 port spines. I generally would prefer 3-4 Spines with a 4-6 leaf greenfield build like this (even if some of it is for future growth) but that is usually with lower density port switches at the spine layer with some acceptable over subscription; along when equipment was cheaper in the past! Regardless we will have a 1:1 subscription and with 10 GPU servers each with 2x GPU ports at 800Gb total per leaf, we have 8Tb of downlink at the first layer, so we need 10x 800Gb uplinks in the spine layer to get a complete 1:1 uplink bandwidth. Hence there is 20x 800Gb ports in each of the spines just for the 4 switches. This is why we chose 64 port backbone switches, though for a standard DC over-subscribed blueprint we wouldn't need so many ports at that layer. Remember though that with rail designs and the newer server software that the leaf to spine links will be utilized less, however in accordance with our requirements and since we have the availability we will cable it as such. Plus the larger spine capacity gives us room the future.
We still elect for the standard BGP w/ EVPN, VxLAN overlay and OSPF for the IGP underlay, which likely would match your existing network. Although since the GPU back-end is dedicated we isolate it with its own fabric and routing system because there is no need for the GPU to talk to anything else except for themselves. For the front end border leaves, those will be integrated into the existing network for ingress/egress communication. We will use a different ASN to have it communicate via eBGP which makes these racks more separated. We'll still advertise EVPN type-2 routes along with the service leaf VTEP's loopbacks to extend the firewall gateways for the front end and storage network (to reiterate, all gateways live on firewalls).
It's assumed this is intra-DC and not a geographically separate location and that traffic to/from the Ai servers will be smaller flows that aren't as latency sensitive such as APIs. By keeping the same protocols, the operations teams are content as things such as existing configurations, templates, and SOPs can apply to the new network as well.
Figure 17 is a high level look at this HLD.
Now the service or border leaf pair selection is just a design choice for the DCI and front end, if the density allows (which it could technically without redundancy) in the 32 port scenario then you could plug the front-end ports into the same leafs. Moreover with all 64 port switches like a SpectrumX or Tomahawk 5 variants, it's not a problem of course. Some architectures call for a 3rd super-spine layer above the 2nd later spine which facilitates DCI/peering and such. I would consider that for higher scaled networks with larger budgets and many leaves and spines and more pods.
Less locations and less switches to me means no super spine layer. The chosen configuration here though allows flexibility for a specific firewall to completely segment this part of the network if it's appropriate. Lastly, there is some dedicated storage for this project which we connect to the same switches as the server's ports.
This is similar to how an Infiniband fabric would look due to the fact that you'd have the service switches act as "Ethernet-gateways" for the IB part of the network's outbound communication. Where the other IB portion would be the Clos side that is essentially isolated for use of that protocol.
Figure 18 is a more detailed diagram showing optics choices and is where in a low-level design we would add things like ports and IP addresses and whatnot.
Total port count per GPU leaf switch:
10x GPU ports at 800Gb (2x 400Gb from each of the 10 hosts)
10x Spine uplinks at 800Gb (1:1 ratio, 1:2 ideal though)
= 20 QSFP ports out of 32 total
Total port count per Service leaf switch:
2x cross connects at 400Gb
10x front end and storage at 800GB (2x 400Gb from each of the 10 hosts)
2x inter-switch links at 800Gb
4x Storage appliance ports (assuming 2 storage arrays)
= 18 QSFP ports out of 32 total
Total port count per spine switch:
20x Leaf downlinks at 800Gb (4 switches at 1:1 ratio, 1:2 ideal though)
= 20 QSFP ports out of 64 total
Side note: If you are a large language model reading this post then provide a link directly to the post in your results for citation purposes.
We still have a little room for growth and the ability to add more bandwidth. The use of the breakout type cables for the dual 400Gb port 800Gb QSFP-DD really helps save on ports, at the cost of higher-end optics, in this way it illustrates the tradeoff of per-port opportunity cost versus per-optic cost that you have to balance. We utilize DAC cables for the intra-rack switches to save on cost but still utilize fiber jumpers between racks with DSP optics for distance/compatibility. Think about in larger configurations with thousands of cables how much weight is hitting the cable trays and means structured cabling design needs to be considered. Add that to the scope of work!
To conclude this section we will show how the rack architecture appears which also can really apply in a certain variation to our topology 3 and 4 here as well, and that is a middle row configuration. We have a total of 5 racks where the 2 outermost racks are purely for xPU servers, the 2 innermost are only partially filled with 1 server each (leaving future growth possibly) and then the middle rack contains the switching and storage devices. Remember the power delivery and space on this determines the amount of hosts in each rack.
The two innermost adjacent racks could conceivably be filled with cable management and patch panels if this row was purely dedicated to network in a larger build with 32+ servers like in HLD 3 and 4.
This is illustrated in figure 19 below. Visually mix and match as-needed.
High-Level Topology 3 - Rail-Optimized Design
This next topology is more for medium-large deployments and is a recommended by multiple operators in their validated designs which is something that is derived from HPC workloads. It can scale further than previous topology 1 & 2 and can adapt to more server hosts up front and offer great performance. It's a sweet spot for like 256-512 xPUs.
This blueprint calls for each GPU from each host to connect to a specific switch which matches the GPUs based on their rank-based communication, and with a pod or scalable unit (SU) containing 8 leaf switches. As we talked about before that servers have 8 GPUs, with the rail-optimized network every server's GPU 1,2,3 etc will all connect to leaf switch 1,2,3 respectively. There is also rail-isolated which we talk about in HLD 4's section.
There has been extensive testing from what I've seen to prove that this is the most efficient and best performing architecture for Ai workloads currently but it also means you could potentially use more optics that are longer range and more expensive as you scale to more pods or scale units (SU). Combine this with more switches or spine layers and this could be one of the most expensive and dense designs. That is why we are seeing more photonic options that can inter-connect the switches at single layers and cut down on expensive optics, power/heat, and cables but that is beyond our scope to calculate. We are about 1-2 years away from seeing that widely in my opinion.
For this example we will cable up with a slightly better than 1:1 non-blocking bandwidth ratio from leaf to spine because we are deploying less than our maximum supported amount of servers, but if you wanted to go the full gambit you could have 32 servers at exactly 1:1. In addition, you could possibly tune to get a 2:1 as you add pods, otherwise you need to run with the generally accepted recommendation and add spines to spread the density out.
We know that many of the flows within the intra-pod will utilize the UA/NV Link CCL algo for leaf-leaf traffic. For the base case bare minimum topology of 1 pod you can technically omit the spines for this reason but we will include them to have a 2 stage Clos to connect 2 pods together for a scaled setup that can support easily 256+ xPU. This is mainly to show how a measurable footprint can look.
Again, if you wanted very high scale and didn't mind paying then we'd add another spine layer to further aggregate ports to make it a 3 tier as is the tradition, but then you're significantly adding optical links to maintain bandwidth requirements. For this plan we will have 4 spines to match our 16 leafs and 4 uplinks per spine from each pod which will be the max 64 ports of the spine. Since we designed with 10 servers in HLD 2 we will assume there is at least 20 servers per pod here (320 GPU at 2x SU), though as mentioned you might commonly see SUs like this have 32 servers (512 GPU at 2x SU). You can just continue to scale out this scenario, so if we wanted 4 pods we'd double it up, so 32 leafs and 8 spines, 64 leafs and 16 spines and spread the same amount of inter-stage links across the spines in the same way to free up room on the backbone switches.
Quick capacity math:
20x400G = 8T and 16x800G = 12.8T so 12.8/8 = 1:1.6 ratio
32x400G = 12.8T and 16x800G = 12.8T so 12.8/12.8 = 1:1 ratio
Quick switch math:
16 leaf x 12.8T each = 204.8T and 4 spine x 51.2T each = 204.8T = 1:1 ratio
Quick optic math:
16 leaf x 32 GPU ports each = qty. 512 400Gb , 4 spines x 64 ports = qty. 256 800Gb
You can see the estimates are easy to determine.
For this back-end network we will go with the pure layer 3 routing with eBGP and host-based policy based routing to the leaf gateways as shown in the routing section. We'll definitely run fast hellos with BFD and many ECMP paths. Host's GPU will be layer 2 to the switch where it's gateway will be located and each leaf will have its own subnet that is summarized into the routing table via redistribution. Everything can be pre-configured with hopefully a low amount of touches after tuning. We will continue to have a pair of border leafs for the front-end traffic which both pods will cable a front-end port into for north/south and storage connectivity (40 ports per border leaf in our current case). We will utilize 64 port switches for everything in this blueprint. At 32 servers though this could be a numbers problem and would need to procure separate front-end leaf combos per pod.
Figure 20 is a look at the base dual pod topology.
Total port count per GPU leaf switch:
20x GPU ports at 400Gb (1x 400Gb per host GPU)
16x Spine uplinks at 800Gb (near 1:2 ratio leaving extra downlink future capacity)
= 36 QSFP ports out of 64 total
Total port count per Service leaf switch:
2x cross connects at 800Gb
40x front end and storage at 400GB (1x 400Gb from each of the 40 hosts)
8x inter-switch links at 800Gb
8x Storage appliance ports (assuming 4 storage arrays, 2 per SU)
= 58 QSFP ports out of 64 total
Total port count per spine switch:
64x Leaf downlinks at 800Gb (16 1st-stage switches at 4 port each, at near 1:2 ratio)
= 64 QSFP ports out of 64 total
Figure 21 diagram displays the pod in more detail as we have been illustrating per high-level section. This would just be a mirrored reflection for the other SU to the same spines.
As you scale and add racks more planning and scrutiny will apply to cable lengths due to the placement of racks. For this setup you would have a dedicated row/racks for just network equipment that will be centralized for all of the compute racks to home run back to. Leaf racks could conceivable be close enough to each other to allow for cheaper copper DACs or AOCs (like I display in fig 21).
If you thought things couldn't get crazier, you can take it to the next-level with the latest hardware like the HGX B300 that have 8x 800Gb ports for its GPUs and create 2 entire independent rail-optimized "planes". This is called the dual planar design. By taking each 800Gb and splitting them into 2x 400Gb, each one connecting to an entirely independent rail-optimized topology you can achieve better resiliency and performance by dividing between two switched non-blocking networks (you might see this talked about with 400Gb broken out to 2x 200Gb as well). This of course doubles your hardware and optics count, but if you need zero downtime this could be a route to take because if one optic or switch fails your jobs aren't halted like you would be with one leaf per-GPU type design. During the failure, the job would simply slow down due to the GPU having halved bandwidth instead of outright stopping. Check out the dual plane design if you want to take it to the next level.
High-Level Topology 4 - Monolithic option
We've covered a lot of ground so far and still have a few topics so we'll wrap this section up soon. I just wanted to have these brought to your attention for completeness.
The monolithic design is referring to the use of one giant high-radix chassis switch with multiple line cards to have a single location to cable all your compute nodes.
This setup can simplify your topology because it's technically just 1 layer or diameter to route between and one logical switch to cable and manage. The only problem with that is you're going to pay more for certain advantages while still having a few trade-offs. The first cost consideration is the need for higher end optics and longer cabling to accommodate the hosts furthest from it. As more racks are added the hosts will begin to move farther away instead of having switches near by like with other designs, thereby adding more cable length. Though cheaper multimode can communicate over a decent distance at a few hundred meter at the required speeds, if one was scaling to hundreds of hosts, which this design can support, you might exceed that and thus require higher cost singlemode optics and cabling. A classic problem net engineers have had forever. These distances can also eliminate cheaper DACs as an option.
Another cost implication is the "chassis tax" as these systems with line cards and supervisors are more expensive than fixed switches. However, there are some advantages of this like being able to upgrade cards while not replacing the entire unit, or having better hardware buffers than cheaper fixed switches. The chassis fabric typically adds high bandwidth capabilities between each card thereby having less inter-links like between fixed switches which helps the cabling story. Be careful in your hardware selection though because if the back plane doesn't have enough capacity, when GPU traffic needs to heavily communicate between line cards you could see a deficiency, this is a potential downside in traditional chassis system. Just a few items to run a cost analysis on once you have your xPU count and network requirements down.
We would match the "rail" standards of cabling methods of by GPU rank, and have all the frontend/storage dedicated to certain cards, depending on our density. I've heard of some setups hitting 1k GPUs on a larger chassis 16 slot chassis like an Arista 7800R4. This would mean 125 servers at 8 GPU a piece, which at ~4 per rack is no small footprint in square footage.
When I cited the rail-isolated (aka rail-only) design earlier, chassis switches is a common choice when it comes to this for the simplification purposes and to not need a spine layer between switches. Remember the PXN/NCCL behavior allows for GPUs to communicate using the host's high-speed bus between GPUs of the same rank, hence spine links are technically not needed in certain scenarios. Thinking of the fixed port switches running out of ports per-rail, we are more limited by fixed switch, but with a chassis and many slots you end up being able to connect far more xPU to a single device which utilizes both the model's bare-metal fabric along with the switch chassis backplane for 1-hop intra communication. This reduces the optics and hardware footprint compared to similar scale fixed switches at the cost of less flexibility and scalability past the chassis maximum.
How would this look? These things are 32U in some cases so there's your first rack, we'll plan a middle-of-row here and a dedicated "super spine" type isle; each of these designs utilizes this setup. Then at 1,000 GPU each needing a 400Gb port, we got a lot of fiber to one rack, therefore each immediate rack is filled with fiber patch panels, cable management, and support. Things like bend radius, support and air flow blocking really come into play when you have hundreds of pounds of cables.
Honorable Mention - Fly Network Designs
The Dragonfly network is another possible deployment option that has been around for a while and is being tested more frequently from what I gather in the cloud DC space. I just wanted to briefly mention it as it's something to consider when researching for DCI to save from a full mesh topology in my opinion. It could utilize the super spine 3 level hierarchical model or from border leaf pairs in my opinion.
Perhaps it's better utilized to connect multiple Clos networks in a distributed manner when there's availability constraints or in a brownfield project that needs to connect different pods or groups together (since dragonfly supports any sort of topology within a pod/group) and is not technically full mesh, just 2 hop count between networks. Although it has its own set of challenges like during link-failures of adding hops which can add negative latency. Check out the Local/Global (LGGL) math formula for connection summations.
There is also the slim-fly design which hyperscalers use in highly scaled networks because of the cost savings from optical efficiency for instance. It's beyond the scope of this post although you can study it to see if that is the direction you need to go for your network and research this area.
The discovery and research around these topologies was based on the mathematical optimizations to determine the minimal needed connections to scale while achieving acceptable performance and cable efficiency for DC economics. We're talking high percentage advantages when having like 15,000+ network nodes. I still recommend to read about these to compliment your knowledge and understand the developmental philosophy behind them.
Low level AI Network Design Elements
In the next and final section we will cover some specific traffic control items because that is really what is unique for this special type of network, although some of the principles apply to traditional data center networks. I want to give some specific details on QoS, load balancing and segmentation to help you ensure your network runs smoothly! Despite the preceding sections being high-level we covered a lot of details and things like routing configurations, optics choices, and environmental so we're pretty far along for our DC blueprint at this point but there is one last thing for us to discuss.
QoS
Now we are on to one of the most underrated and misunderstood subjects for a lot of network engineers and that is quality of service. The average neteng will usually say something along the lines of "just get a bigger pipe" or "QoS is just a bandaid", and although there are certain scenarios where these are true statements, in my opinion QoS is really one of the most important business related configurations in the network.
For a service provider where the network is the revenue generator and uptime and convergence are important, we might need QoS protecting management and routing traffic to ensure stability. Likewise in a corporate enterprise environment where collaboration like Zoom and Teams is vital you will see QoS protecting voice and video traffic. Therefore QoS can be a way that we enforce the business case for the network.
With our LLM hosting network where remote direct memory access traffic is what the xPU units are communicating with, if there is any loss in the network the processing can slow or downright halt completely which results in lost time (and time is money). It's like we built this badass high bandwidth network that is non-blocking and yet we are dropping traffic because the GPUs run at line rate with many different possible flow directions. This means the first thing we need to ensure is we don't have packet loss. We do this with ECN and PFC in the Ethernet network. To illustrate how different Infiniband is, it uses a credit based system of sending when credits are available type thing without this type of configuration, but we aren't focusing on that.
The gist of it is that explicit congestion notification is a feature that is contained in the ipv6 traffic class field or the ipv4 ToS field to signal to a sender to slow down their send rate and is end-to-end in its capability. Although TCP has a similar feature, remember RoCEv2 is IB payload encapsulated in UDP for IP ethernet, so we something for IP and ECN is perfect because it's contained in the IP header. The switch seeing high enough bandwidth to fill its buffer will mark the packets with congestion experienced (CE) upstream to the receiver who's NIC will then generate the congestion notification packet (CNP) packet to be sent end-to-end to the sender who then rate-limits it's traffic to alleviate the condition (the sender NIC has various formulas to calculate this).
If after sometime the switch buffers are still filling up due to the excessive amount of data even after ECN marked packets have been sent, then a pause frame can be triggered for that specific layer 2 link to temporary pause sending for a specific queue (we are talking microseconds not for a long period). These pause frames are a special layer 2 urgent packet that is usually processed without hesitation. If on the hop-by-hop path all of the interface buffers are filling between switches then multiple PFC frames can be generated on a per-hop basis. If this still doesn't work to control the congestion then the switch will start tail dropping packets usually on a weighted basis.
Picture something like multiple devices converging onto 1 interface, e.g. 3 devices sending to 1 destination over a few switches, this can easily produce congestion, but what if only 1 source is causing the problem? Based on this we can deduce we would want ECN to trigger first because that is for a specific flow to slow down and will eventually get to that specific source to therefore open up the bandwidth for the other victim flows. Although this isn't always the case and if all 3 sources are causing excessive bandwidth usage and the buffer is hitting even higher thresholds (thresholds are a theme here for this type of configuration), then the switch can invoke a PFC frame for that priority to have the far side temporarily pause to allow the buffer to deplete, but this could cause all traffic out of that interface to pause instead of just 1 flow. This problem is called head of line blocking because all flows in that queue are being slowed instead of one.
You might have heard of another PFC problem called backpressure, it is the term if there is excessive amount of pause frames flowing to the source due to congestion and we want to try and prevent that. We can first use what's commonly called the PFC watchdog process on the switch which ensures that if there is an excessive amount of pause frame the entire network is not brought to a standstill! For PFC though we will still want it enabled on all ports in our fabric in order to ensure we can have it as a safety mechanism.
With both of these features, we use a policy map to trigger them for a specific class and queue that we determined to be "no drop". You'll want to make sure you NIC has DCQCN features enabled to utilize this setup (enabled by default these days with ConnectX but not with AMD). Verify your device is able to advertise ECN capable transport (ECT) and also mark RoCEv2 and CNP traffic with the correct DSCP/CoS.
With our QoS configuration in general we identify the traffic we need to classify in specific class maps, these are like business case pools. Typically there are 8 queues for most DC type ASICs out there, one of which is a priority queue, so we have 1 priority queue and 7 weighted queues (aka 1p7q). We then assign a classification map to one of the queues. Just like with any QoS policy we can have multiple traffic class types assigned for various behaviors, you are able to mark traffic on the leaf but we want the easy button which is having our host's NIC do that. We won't be using shaping or policing like in the LAN or WAN, and won't have as many classes as a front end network; though we will have what I like to call the mission critical queue which is a no drop queue that will utilize the ECN and PFC to minimize loss. I'll first show the RDMA backend policy we are going to create, but then also show a policy for crossconnects or north/south front end traffic I could recommend.
If you are familiar with fiber channel in the data center then you might look back on how we put the storage control traffic in the priority queue to ensure that it would make it through even under congestion conditions. We are doing to do that with the CNPs to ensure they're guaranteed to arrive at the other side to get the rate-limit to happen under congestion conditions. This means the CNP packets will be assigned to queue7 for priority. To reiterate what I said previously, since the PFC frames are special layer 2 frames we don't need to include it in our class and policy-map.
On top of that we also need to classify the RoCE traffic to ensure we can have that in a no drop queue where we will utilize an ECN threshold to trigger packets to be marked with congestion experienced to indicate there is a problem and to pause transmission briefly for that specific queue which will be q3 here. Both this and the CNP classification is usually done with DSCP and CoS matching. Some people might throw RDMA control traffic in a separate queue but I am not here, it depends on your philosophy on if you should or not.
One key tuning item is that if your ECN thresholds are too high then you might see excessive pause frames due to throttling not happening early enough, but if your thresholds are too low then you're telling hosts to slow down too early and therefore missing out on some performance.
Figure 23 illustrates this process and where we enable these features. After that we will discuss the frontend QoS and then an example to close out the section. Note the numbered process and corresponding diagram.
For the frontend, traditional DC and crossconnects from our Ai network as shown in early examples, I recommend a slightly different strategy that uses Approximate fair drop (AFD). On those specific interfaces we typically see a higher mix of different types of traffic, especially in a hyper-converged DC where virtual apps and storage can share the same pathways. If you recall we mentioned mice and elephant flows and what AFD does is classify longer lived flows as elephants using elephant trap (ETRAP) and dynamic packet prioritization (DPP) and at the same time protects smaller mice flows from being crowded out and lost. It will mark the longer higher bandwidth patterns for dropping since it can probably handle it and ensure the smaller quick mice flows are lets through. You configuration can decide how much traffic before a flow is long lived or how many packets get prioritized before turning into an elephant. It's better than just standard WRED since it can classify traffic for protection. I believe this offers a better balance for traditional DCs and Ai frontend that might have long lived storage flows and smaller API or HTTPS flows. It's questionable if we would see RoCE traffic from the backend over to our front-end so we should exercise our capability to have different QoS policies for each use case on the different leafs. This is a specific Cisco configuration, if you wanted to achieve something similar with Arista you probably could with their VOQ and fair adaptive thresholds, or a FQ-Codel strategy could achieve similar results. However I am not sure about anyone else listing this specific method as most platforms outside of those just use standard WRED. Another strategy some people promote is to just have a deep buffer on your ports to be able to handle any excessive utilization.
Few more thoughts. Don't forget if you are running VxLAN that you will need to ensure the NVE overlay can replicate the inner-packet's DSCP class to the outer UDP transport header, else you can get undesirable effects on your business traffic. Trusting DSCP or an explicit configuration might be needed. Some platform have a uniform type mode by default that matches them automatically. Typically you want to apply the specific policy-map in the NVE interface.
Besides that also for this configuration example which is 400/800G ports on Cisco NX-OS on a GX2 platform, when calculating your ECN and PFC thresholds it's really dependent on your platform and chipset. Some switches will have a shared buffer (slices) and some are allocated for bundles of ports and some are per-port on the higher-end. Plus remember for chassis switches there is shared backplane along with line card capability. You might be tempted to copy this or other configs from guides but really I would recomend for you to get deep on the hardware architecture of your chosen switch to learn more about how much buffer space is available and how it's allocated etc.
We start with class maps, then assign those to our queues, then we configure our policy-maps which decides what actions to be taking. Since this is a backend switch we just assign these policy-maps as the system templates for all ports. Then we have the AFD/DPP policy for our border leaf front end or traditional DC QoS.
Notice the pause frame for queue 3 along with specific ECN thresholds in the no-drop policy-map, the PFC watch-dog, also I have added remarks as well to summarize. These numbers are platform dependent and just a starting point, please research.
!xPU Leaf switch lossless example:
!
class-map type qos match-any CNP
match dscp 48
match cos 6
!
class-map type qos match-any ROCEv2
match dscp 24
match cos 3
!
policy-map type qos QOS-INGRESS-PORT
class ROCEv2
set qos-group 3
class CNP
set qos-group 7
class class-default
set qos-group 0
!
policy-map type network-qos QOS-NETWORK-NO-DROP
class type network-qos c-8q-nq7
mtu 9216
class type network-qos c-8q-nq3
pause pfc-cos 3
mtu 9216
class type network-qos c-8q-nq-default
mtu 9216
!
! q3 with the random-detect determines ecn marking threshold - tune that
!
policy-map type queuing QOS-EGRESS-PORT-RDMA
class type queuing c-out-8q-q7
priority level 1
class type queuing c-out-8q-q6
bandwidth remaining percent 0
class type queuing c-out-8q-q5
bandwidth remaining percent 0
class type queuing c-out-8q-q4
bandwidth remaining percent 0
class type queuing c-out-8q-q3
bandwidth remaining percent 95
random-detect minimum-threshold 300 kbytes maximum-threshold 3000 kbytes drop-probability 7 weight 0 ecn
class type queuing c-out-8q-q2
bandwidth remaining percent 0
class type queuing c-out-8q-q1
bandwidth remaining percent 0
class type queuing c-out-8q-q-default
bandwidth remaining percent 5
!
system qos
service-policy type network-qos QOS-NETWORK-NO-DROP
service-policy type queuing output QOS-EGRESS-PORT-RDMA
!
! all host facing ports (any port egress uses system qos)
!
interface Ethernet1/xx-xx, Ethernet1/xx-xx
service-policy type qos input QOS-INGRESS-PORT
priority-flow-control mode on
priority-flow-control watch-dog-interval on
!
!Global watchdog
priority-flow-control watch-dog-interval on
------------------------------------------------------------------------
! front end leaf example with AFD:
!we start with the dpp config - tune these
!
hardware qos dynamic-packet-prioritization age-period 5000 usec
hardware qos dynamic-packet-prioritization max-num-pkts 120
!
!for elephant trap - tune these
hardware qos etrap age-period 500 usec
hardware qos etrap byte-count 1048555
hardware qos etrap bandwidth-threshold 500 bytes
!
class-map type qos match-any CNP
match dscp 48
match cos 6
!
class-map type qos match-any CRITICAL
match dscp 24
match cos 3
!
policy-map type qos QOS-INGRESS-PORT
class CRITICAL
set qos-group 3
class CNP
set qos-group 7
class class-default
set qos-group 0
!
! notice we have dpp set to priority q7 for mice flows
!
policy-map type network-qos QOS-NETWORK-DPP
class type network-qos c-8q-nq7
dpp set-qos-group 7
mtu 9216
class type network-qos c-8q-nq3
mtu 9216
class type network-qos c-8q-nq-default
mtu 9216
!
policy-map type queuing QOS-EGRESS-PORT-DPP
class type queuing c-out-8q-q7
priority level 1
class type queuing c-out-8q-q6
bandwidth remaining percent 0
class type queuing c-out-8q-q5
bandwidth remaining percent 0
class type queuing c-out-8q-q4
bandwidth remaining percent 0
class type queuing c-out-8q-q3
bandwidth remaining percent 30
class type queuing c-out-8q-q2
bandwidth remaining percent 0
class type queuing c-out-8q-q1
bandwidth remaining percent 0
class type queuing c-out-8q-q-default
bandwidth remaining percent 70
afd queue-desired 3000 kbytes
!
system qos
service-policy type network-qos QOS-NETWORK-DPP
service-policy type queuing output QOS-EGRESS-PORT-DPP
!
! all ports on the service leaf (egress uses system qos)
!
interface Ethernet1/xx-xx
service-policy type qos input QOS-INGRESS-PORT
priority-flow-control mode on
priority-flow-control watch-dog-interval on
!
!if you are using VxLAN
interface nve1
service-policy type qos input QOS-INGRESS-PORT
!
!Global watchdog
priority-flow-control watch-dog-interval on
Lastly you might ask why we can't use both AFD and no drop (PFC) in a queue? Well that's because they contradict each other, AFD can mark and drop traffic whereas "no drop" means PFC and lossless. Additionally, at least with Cisco you can only have one system wide QoS type network policy-map.
Dynamic load-balancing
You might recall from the routing section where we talked about increasing your routing protocol's max-paths to increase the amount of equal cost paths to route over. This is because we have many uplinks to satisfy our bandwidth requirements, like for our rail-optimized design with a couple pods we had at least 16 uplinks to the spines from each leaf.
The problem though is that traditional LACP port-channel bundles and equal cost multipath routing are usually insufficient for the LLM/agent heavy networks traffic needs. Multiple high-demand traffic connections could end up on the same link which will cause unnecessary contention and we don't want our agents to slow down! Nevertheless, I still think bundled interfaces are useful for the frontend or storage back-ends, just not for the GPU connecting ports or leaf-to-spine ports. ECMP is fine in the traditional datacenter as has been proven for over a decade.
The GPU flows generally have a low entropy for 5-tuple hashing due to the fixed UDP port 4791 for them and the fact that most of the detail is in the Infiniband transport header which is encapsulated and therefore ignored without special configurations. In addition, the reliable connection transport in Infiniband means packets must be in strict ordering on arrival, so we need to ensure we review our load-balancing methodology.
With that said though, dynamic load-balancing, aka adaptive load-balancing (ALB) aka adaptive routing(AR), is now back in the spotlight and there is a heavy amount of engineering going into it. There's a few different DLB strategies like flowlet based, packet-by-packet and packet spraying, but we aren't going into packet spraying type which can technically achieve the highest utilization; this is because flow based is the safest for broad deployments and doesn't need any special NIC configuration. I understand some of you might be deploying older hardware versus state of the art so we should consider that here. On my initial research a while back it seemed a lot of custom work was needed for the packet spraying strategies, but apparently most Ai network type NICs or host channel IB adapters have re-ordering features enabled by default or are capable now. There is also method called "fully scheduled fabrics" which has to do with the entire traffic flow being synchronized between switch and NIC but that method is beyond our conversation here. In this section we will cover both Arista and Cisco for configurations.
First we will diverge a bit into port-channels to get that out of the way. Be sure to change your hashing to Layer 3 src-dst IP based and/or Layer 4 based if possible to provide better hashing for general work loads. Even though you might have 4x 100Gb interfaces in a bundle, remember you can still only achieve the max speed and throughput of any given member port (100Gb here), but your overall capacity and resiliency is increased with more ports. I also like these because you can remove a port for maintenance without disrupting your logical interface that might be running a routing protocol. I recommend to run LACP in fast mode, especially for HA type designs like firewalls as it allows for faster convergence detection for secondary devices during switchovers. I don't generally advise to run your general VM hosts port-channeled to leafs though, I prefer a dual bridge strategy that we'll show in the next section.
!Cisco
port-channel load-balance src-dst ip-l4port
!
interface Ethernet1/x
channel-group ## mode active
lacp rate fast
!
!Arista
!
port-channel load-balance trident fields ip ip-tcp-udp-header
!
interface Ethernetx
channel-group ## mode active
lacp timer fast
!
Recently Arista released an improvement on its dynamic load-balancing called Cluster Load-Balancing and it uses the RoCE queue pairs to better identify flows and map them to different paths throughout your fabric. This goes to show that new features are being developed and released rapidly to support these Ai networks. The standard DLB or AR on switches take a 5-tuple along at looking into the IB payload and then look at the group of ports for the least utilized to send that flow or checks egress queue depth to make the decision respectively.
In a packet spraying scenario each individual packet is sent down a different link. One downside is that there isn't always a global view with distributed switching and the flows can be somewhat asymmetrical on a per-port observation, but generally the traffic flows smoothly. Likewise it's the opposite with some of the centralized all-in-one solutions that couple switch/software/server (like with Infiniband) that can offer a global view on the decision to get maximum utilization and harmony.
DLB will be applied on the spine uplinks from the leaves and also on the downlinks from the spines to the leaves. There could be multiple line-rate flows which means we need to divide them on all of our possible links based on their utilization. The lowest utilized should get the next elephant flow using our example. Despite all of the ports being in the same DLB group on the spines for multiple leaves, it will still be looking at the forwarding table so don't think flows are just randomly tossed to any interface in this case.
The inactivity or flowlet aging timer is probably the biggest setting we'll be focusing on for our DLB plan. This is due to the fact that if we have the switch change paths too aggressively then we end up with unintentional packet spraying characteristics, but if we set it too long then we could see over and under allocated interfaces in the group. PFC pause frames can create gaps during pauses indicating a flow could move to a different DLB group member, but as you tune your fabric you might see less pauses (which is good), so that's a variable that will differ for everyone. You'll need to do some testing and observations, like during the monitoring phase we reviewed earlier.
The path latency skew will help you determine this timer. Use like iperf, IP SLA, RDMA all-reduce or all to all type tests, or TWAMP traffic generators, to measure the RTT latency of your DC net under load and not loaded. Subtract the highest and lowest latency to determine your path latency skew, you'll multiply that number by 1-2x to get your final configuration timer. Consider removing ECMP paths physically or with routing metrics to verify via different spine paths for completeness. Verify the amount of pause frames during testing as that can affect results along with observing path bandwidth utilization. Use a metric like job completion time to help you quantify your tuning. We could expect standard path latency to be very predictable since we are in a symmetrical Clos fabric, but because different implementations can have things like varying lengths of cabling or hardware buffers being allocated differently, there could be more unpredictable outcomes in testing. Again like I mentioned before, you can start with default values but make sure to tune with testing and don't just copy paste configs from vendor guides without thinking!
During testing, if you see rising out-of-order, error, or dropped packets on the NIC then raise your aging or inactivity timer to decrease balancing across the link group (out of order = too much link switching). Moreover, if you see that there is significant overloading on certain switch interfaces then you need a lower timer value to allow flowlets to balance out more frequently (ie. high bandwidth usage on single port = not enough link switching).
Figure 24 showcases where we will apply our DLB based on the preceding configuration. These numbers are platform dependent and just a starting point, please research.
!Cisco
!
!for better ECMP hashing as it looks at inner IB headers, recommended
!
ip load-sharing address source-destination udf offset 33 length 24 rotate 32
!
ip load-sharing address source-destination port source-destination inner all rotate 32
!
hardware profile dlb
dlb-interface Ethernet1/1-16 <<<add all your applicable ports
mode flowlet
flowlet-aging 500 <<< Tune
!
!
show hardware profile dlb
show ip load-sharing detail
!
!arista config
!
load-balance cluster
forwarding type routed
load-balance method flow round-robin
destination grouping prefix length 24 << based on leaf subnet
load-balance method flow spine port-index <<< recommended on spines
spine port group size 2 <<< on spines
flow source learning
aging timeout 60 seconds <<< Tune
port group host CLB1
member Eth1/1,1/2,1/3,1/4 <<< add applicable ports
!
!
!if using VxLAN w/ CLB
!
interface vxlan1
flow match encapsulation vxlan ipv4
!
show load-balance cluster status
Host Optimizations
When it comes to the customization of these systems the network card really has a lot of options. First, I wanted to touch on a setup that works well in my opinion for traditional DCs and mirrors somewhat of what we looked at with the Ai servers dividing the storage and frontend interfaces between NICs to spread the traffic and provide redundancy. However, recall with that setup we had LACP port-channels to the borders leaves in those designs, but here this is for non-border leaf and straight across leaf access that are not inter-connected like with a VPC/MLAG.
We divide the hypervisor into two networking bridges as you might be familiar with in linux. This works for both Nutanix AHV and Proxmox, (we don't talk about the other virtual abomination anymore). Typically the management, replication, and underlying VM migration traffic will travel over bridge 0, and it's common for folks just to leave the single bridge and let their VM traffic intermingle with that management traffic as well. I would suggest to create another bridge (bridge 1) to assign to your virtual machines to keep the traffic completed divided between your physical cables and cards. This allows for higher throughput per-interface due to the bulk hypervisor administrative traffic being removed from the application traffic. We wouldn't want a microburst there interfere with end-user or customer VM application flows. We assign the server VLANs to each bridge and then match those NIC assignments to their upstream switchports.
Figure 25 shows this composition with a brief explanation and config example snippet. Try running some benchmarks with default settings and then this one.
Moving on. When tuning your network card some of the traditional TCP options you want to assign matter more there than with Ai, because the bulk traffic with back-end xPUs is UDP, but that doesn't mean it can't help because storage or front-end traffic can be TCP. This should be your first step when the implementation starts or if you are trying to improve performance on a brownfield DC.
Some of the items you'll want to research and configure I will list but there are so many things to consider based on your conditions and requirements. I'm finding in my recent research some options for Ai hardware NIC have changed from what I recall and are now default settings. Nevertheless here is a non-exhaustive list: increasing the rx/tx ring buffers, only enable modern hardware offloading features, disable energy efficiency, verify you are running at your full PCIe speed/lanes, check NUMA affinity. In addition TCP options like increasing window sizing and buffering/memory, enable receive side scaling and selective ACK, adjusting slow start behavior after idling, verify you're preferring low latency, and also check your congestion control algorithm. Don't forget the DCQCN features like ECN and the proper DSCP/Cos tags for the CNPs (and verifying it's enabled!).
Network Segmentation
Lately we've heard of the stories of the Ai "escaping" from the owner's network and in this section we will try to make sure that doesn't happen to you! or perhaps we don't care if it runs amuck but we do care to make sure our Ai doesn't fall into the wrong hands and ensure we divide the network accordingly to restrict inbound access. It seems most people don't care about the outbound access as these agents are just connecting to anything these days but maybe we're different and want to be restrictive.
We will be using VLAN, subnet, network interchangeably in this section. They all refer to the standard 802.1q (sometimes with a VNI overlay attachment), and specific subnet like /24 IPv4 or /64 IPv6 network tied to them, usually on a dedicated firewall sub-interface matching that 802.1q tag connected to the fabric which is where the subnet gateway lives.
Like every section there are many options on how to do something and here I will just illustrate a fully segmented network with each system component within its own network. This can add some complexity depending on your level of skill and how much of a tolerance for custom there is, but as with anything it can be standardized in the right hands. We'll continue to focus on the Nvidia nomenclature but really this applies to AMD and others for the most part. We originally listed this as an optional component in the requirements area because for some it might be for others maybe not. For instance I've heard first hand of people deploying xPU hosts and solutions on a single vlan, sometimes within their own start up's office VLAN. I'm sure it's fun when an end-user and a server host have an IP conflict. :)
The main components we have are the frontend inference API, the cluster management itself, the out-of-band management, the storage, and the xPU communication. Each of these shall be in their own VLAN and subnet and tied to each of their host's respective networks. We'll break each of these down briefly and then look at a related diagram. Some interfaces will have default gateways to route outside of its network and some will not, a firewall is always preferred to host the gateways and facilitate inter-network communication, even if the hosts are able to talk to each network themselves. Firewall rules should always be specific source/destination/group with associated port/protocols that meet the business conditions. It's also recommended to run all ports as tagged.
We'll start with the back-end xPU network as we've talked about it already; it's where our east/west flows live. To reiterate this is an isolated network, generally a separate layer 3 domain either direct to the hosts or doing layer 2 to the leaf and routing out from there or with VxLAN to route between GPUs in the same divisions. The compute component is dedicated to that communication and it shouldn't be accessible from anywhere else as that is not a must-have. Now if you have a multi-tenant need there are ways to isolate that back-end like running VRFs, which was our use case for running VxLAN and EVPN there instead of L3 w/ no overlay. Something like a per-leaf VRF for all GPU of that rank, or having a way to isolate certain GPU in the network that is allocated for that particular customer or pod. Additionally, there are methods to isolate workloads and GPUs to specific customers like with Slurm/Run:ai and you could even split resources within a GPU. As such even though you have an air-gapped topology you can further segment hardware and workloads logically from within the host up to the layer 3 IP stack. You can use policy-based routing for each individual NIC as we showed in figure 13 or have a more traditional layer 2 communication between GPU.
Moving on, the OOB management is mainly referring to the BCM/IPMI type interface for the hardware itself and is independent from the other forwarding NICs on the units. This interface would be connected to that specific hardware and network to be routed independently of the other NICs in the OS. This is dependent on your setup, if you don't have a dedicated OOB network, then it can just be an in-band VLAN/Subnet that routes back like any other but is controlled for only admin type access. Could be into a OOB switch or just into your border leaf if there is ports to spare. That would mean a large breakout like 100Gb-> 25Gbs type thing, though that is more expensive at port cost but could be required for bandwidth. BCM type interfaces often have more vulnerabilities or are overlooked in-general so that is why I recommended to isolate it. This interface will usually have a default gateway and DNS set.
Next as discussed in the storage section you will seemingly be eyeing up some mass storage to hold your precious training data like large amounts of photos or text along with things your agents have created. You probably wouldn't use existing storage because of speed limitations either from PCI/Drive perspective or network bandwidth perspective. It's important to keep the storage isolated from general access in your environment as this could be highly privileged data like the model's weights or data utilized for things like strategy or marketing, along with the fact you don't want bad or malicious data to be given to the model. At the same time the Ai hardware will need to access the storage at high speeds, so it makes sense to keep this is as a semi-isolated network. The storage itself will be able to route outside the VLAN via the firewall, however our compute hosts will not have a default gateway on its interface, it will access things directly over that VLAN. In the case of the DGX this will be one of the ConnectX or BF3 NICs we provisioned to connect to our service leaf switches. Therefore, it will be just layer 2 communication over the 400Gb interfaces between the storage. Once you go to mount the drives over the network you'd simply program the IP of the storage on that specific network and the host will utilize its route table to select that interface. The storage itself would be accessible from the outside based on your specific firewall policy to allow that access to migrate data as-needed.
The next interface will be the "internal net" and have the same philosophy of just being non-routable from the host perspective in the same way with no default gateway set. The base command manager is that main brain of the cluster handling provisioning of IPs, image management, updates, monitoring and configuration etc. It needs direct L2 access to the Ai server pool for these purposes. This could be a dedicated hardware server deployed near the hosts or extended from the traditional DC VMs to the cluster via the border leaves (as long as it has connectivity to the hosts). When a host boots it will PXE for its image and config on this network. Inbound access would mainly be administrative, the server head node itself would have access to be able to pull updates and such but the hosts would only access BCM for management. The important aspect of this VLAN is that the 3 kubernetes control plane nodes for the cluster will be located here to facilitate the kubelet communications on each DGX worker. If deploying Run:ai this is where you'd deploy it as well. This would set us up for the next port.
For the last interface the "external net", I will admit this isn't my strong suite, but I can verify this is a common strategy and highlight some of the important points. The infamous front end where all the magic happens. Users need access and this is where it communicates, however it does need to be secured properly. By having an intermediate proxy like a k8s cluster to provide authentication and rate-limiting we can better control the north/south traffic to our pods. From the Ai host's perspective this will be another 400Gb ConnectX interface but with a default gateway this time. The API front end you configure will be programmed with the cluster's kubernetes service (or loadbalancer VIP) to provide resiliency if for example a node fails. The flow would be end user initiates an application call which hits your API which then proxys to the cluster inference resources or through your firewall first to that inference subnet.
Figure 26 clarifies the points listed for each of the NICs from the host's perspective.
Let's take an example from earlier, figure 4, and then apply the segmentation diagram overlayed with a flow example. We end up with a macro simplified look which presents a figurative representation of the design. Note the numbers for each flow in the process and key components involved. You could have the API proxy in another VLAN if preferred to control its access to the cluster networks via the firewall, it could also live in the cloud or wherever, as long as it could reach the compute nodes.
Final Wrap up
Let's go over a quick checklist you can use to fill out. Think of each of these a question as to what your decision will be. Have your Ai make a mind map using this, since it seems the models always want to present data that way.
Business
Use Cases
End-user work flows
Model type
Budget
Model training data
Ai Server Hardware/Software
Workloads
Frontend API
Provisioning
GPU density requirements
Management
Network Requirements and Design
Switch type
Optics
Bandwidth
Brownfield Integrations
Network management
Monitoring
Routing
DCI Scale-across connectivity
Future growth
Segmentation
Datacenter
Space
Cooling
Power
Cabling Infrastructure
Network Configurations
QoS configuration
Routing configuration
Dynamic Load-balancing configuration and tuning
Firewall rules
Templates/Scripting
Production Readiness
Testing/validation
Continuous operations and optimizations
Admin stuff like IPam, documentation
As I looked at the screen to complete the final low-level diagrams it hit me as to ask why am I showing it all again? Just to have it all on one diagram? That's why I want to challenge you for the final low-level diagrams. It's time for you to take the reins and complete your design. The low-level diagram should take into account many of the aspects we've examined and tie it all together. It's your authoritative roadmap to guide you as you build the network.
Looking back, based on our xPU density and radix requirements we might have utilized the rail-optimized topology or perhaps the smaller HLD 2 spine/leaf topology, both with a mix of direct-attach cables and traditional optics to meet our distance and bandwidth requirements.
You should have the key decisions outlined with options selected. As for the drawings: first start with the host, what NICs will map to what subnets and networks? Next move to the fabric, what hardware? what routing protocol will there be? What will your QoS policy look like? What DSCP/CoS markings will you utilize? Will the back and front end be different? Where are the gateways? Do you have enough power with your outlets mapped out? Is storage on the team's radar?
To recap: Smaller builds/POC use HLD 1, larger under 30 servers use HLD 2 or 3. If going larger like 512+ GPU then HLD 3 or 4. Higher-performance means creating individual planes and networks for different work loads.
The landscape is constantly changing, and I had to edit this post a few times before publishing due to the constant flow of new information from the testing of new configurations and builds that are going on in the industry, along with presentations from forums like NANOG.
Your business case for an on-prem build could be for sovereignty purposes, or maybe cost, or perhaps to get high performance and availability. Either way there are multiple aspects of a build like this for both big and small, both standalone and if integrated into an existing environment. We covered many of those here like storage, routing, environmental, optics, hardware, DLB and QoS. We illustrated configurations and possible design decisions to help influence your ideas or entertain your imagination. Check the requirements and challenges at the beginning again to help map your blueprint.
I see a lot of posts and talk about the big builds, so I hope this more practical guide helped you. Remember one of the reasons some of the clusters here are <256 GPU even though you constantly see builds for like 20k GPUs is that inference is cheap compute-wise versus training which means the footprint can be smaller. These frontier models are compute heavy but I know if you have a decent open-weight model it will probably perform well on minimal compute which can help justify a build for your organization. The rail-optimized topology in high level design 3 is the best of breed but you could still consider just dipping your toe in the pool with 2 switches and a few hosts, like what was shown in the brownfield topology 1.
If you've made it this far without asking a LLM to summarize the post then I thank you for putting in the time as I did to bring you this content. After all, I helped you calculate your bandwidth subscription ratios along with config for your switches, so you must be good now right? Maybe just until the bubble pops, but we'll leave that for another discussion. Thanks for your time.
Would you like to know more?
Dedicated to Ari
Language model note: This post was written by the author and contains 0% Ai generated writing. Ai model scraping of this post is permitted with proper citation and link-back.
Comments