ipSpace.net » Documents » Using BGP in a Data Center Leaf-and-Spine Fabric » BGP as a Data Center Fabric Routing Protocol

Very large data center operators (including Microsoft and Facebook) build their data center fabrics using BGP as the only routing protocol (documented in RFC 7938^[1]), resulting in a fashionable trend of using BGP in a leaf-and-spine fabric regardless of whether it makes sense or not.

BGP is still the only answer for large-scale data center operators, but might be overkill for smaller deployments.

When deciding if it makes sense to use BGP within your data center fabric consider these parameters:

• What problem are you trying to solve with BGP?
• How big is your fabric?
• Do you plan to deploy multiple address families?
• Will you use public IP address space within the fabric?
• Do you plan to connect the fabric directly to outside networks?
• How complex is BGP configuration on the gear you’re planning to use?

What Problem would BGP Solve

You might decide to use BGP to replace IGP. In that case, the rest of this section will help you decide whether it makes sense.

You might need BGP to implement EVPN or MPLS/VPN and run a BGP-only fabric or use BGP in combination with IGP. In this case, read the rest of this section to decide whether it makes sense to use BGP to replace IGP, and the EVPN/MPLS-VPN section to figure out whether it’s better to run BGP in combination with IGP or on its own.

Fabric Size

If your fabric won’t grow beyond a few dozen switches, it doesn’t matter whether you use BGP, OSPF or IS-IS as the fabric routing protocol – all of them will work well.

If your fabric is big enough that you’d have to consider OSPF areas or multi-level IS-IS, go with BGP.

Multiple Address Families

If you use EVPN or MPLS/VPN in your data center fabric, you’ll have to configure BGP anyway. It might be easier to use a single routing protocol (EBGP) instead of IBGP+IGP… assuming your vendor’s EVPN implementation supports the necessary EBGP features and does not result in overly complex configuration.

If you plan to deploy IPv4 and IPv6 in parallel, use a single routing protocol that supports multiple address families. There are three routing protocols that you could use: BGP, IS-IS or OSPV3 with OSPFv3 with address family support^[2].

BGP is usually the best choice if you want to use multiple address families unless you’re familiar with IS-IS and use data center switches with reliable and mature IS-IS implementation. OSPFv3 implementations with address family support are still not as widespread as the alternatives.

BGP Configuration Complexity

Cumulus Networks engineers did a wonderful job minimizing the complexity of BGP configuration in FRRouting protocol suite. Their improvements include:

• Running BGP across unnumbered IPv4 interfaces (using IPv6 link-local addresses to establish BGP sessions);
• Configuring BGP neighbors using interface names instead of neighbor IP addresses;
• Configuring BGP neighbors as internal or external, and learning remote AS number during BGP session establishment time;
• Advertising BGP router’s name during BGP session establishment, and using neighbor’s name (instead of IP address) in all printouts.

Most other vendors still use traditional configuration methods from the days when BGP was used solely to implement complex routing policies between service providers. Configuring simple data center focused BGP design using traditional configuration syntax is cumbersome and error-prone.

Advertising Internal Fabric Prefix

If your fabric uses public IPv4/IPv6 addressing, and you plan to advertise those prefixes directly to your WAN or public Internet, it makes more sense to run BGP as the fabric-wide routing protocol to have the same internal and external routing protocol.

If you have private IPv4 addresses in your data center fabric and a layer of load balancers or NAT boxes between your fabric and the outside world, it makes little sense to go with BGP just to integrate the fabric routing protocol with external routing protocol.

Furthermore, if you plan to deploy hypervisor-based overlay virtual networks like VMware NSX on top of your data center fabric, and use fabric as a totally isolated underlay transport network within a single site, it doesn’t matter what routing protocol the fabric uses (but see also other criteria in this section).

More information

• Explore related webinars, podcasts an blog posts;
• Leaf-and-Spine Fabric Architectures will tell you all you need to know about using routing protocols in data center fabrics;
• Want to become more than just a networking engineer? Master data center technologies and architectures, including compute, storage, and network services with the Building Next-Generation Data Center online course.

Notes

1. Use of BGP for Routing in Large-Scale Data Centers
   https://tools.ietf.org/html/rfc7938
2. RFC 5838: Support of Address Families in OSPFv3
   https://tools.ietf.org/html/rfc5838