BGP Convergence Optimization

ipSpace.net » Case Studies » BGP Convergence Optimization

A large multi-homed content provider has experienced a number of outages and brownouts in the Internet edge of their data center network. The brownouts were caused by high CPU load on the Internet edge routers, leading to unstable forwarding tables and packet loss after EBGP peering session loss.

This document describes the steps the customer could take to improve the BGP convergence and reduce the duration of Internet connectivity brownouts.

Brief Network Description

The customer’s data center has two Internet-facing edge routers, each of them connected to a different ISP through a 1GE uplink. Both routers are dual-attached to core switches (see Figure 1). ISP-A is the primary ISP; connection to ISP-B is used only when the uplink to ISP-A fails.

Figure 1: Network core and Internet edge

Edge routers (GW-A and GW-B) have EBGP sessions with ISPs and receive full Internet routing (~450.000 BGP prefixes^[1]). GW-A and GW-B exchange BGP routes over an IBGP session to ensure consistent forwarding behavior. GW-A has higher default local preference; GW-B thus always prefers IBGP routes received from GW-A over EBGP routes.

Core routers (Core-1 and Core-2) don’t run BGP; they run OSPF with GW-A and GW-B, and receive default route from both Internet edge routes (the details of default route origination are out of scope).

Solution Overview

The temporary blackout and prolonged brownouts following an Internet uplink loss are caused by BGP convergence issues. Like with any other routing protocol, a router running BGP has to take the following steps to adapt the forwarding tables to link or neighbor loss:

1. BGP routing process detects a link or neighbor loss.
2. Invalid routes are removed from the local BGP, routing and forwarding tables. Alternate routes already present in BGP table could be installed at this point.
3. Updates are sent to other BGP neighbors withdrawing the lost routes.
4. BGP neighbors process the withdrawal updates, select alternate BGP best routes, and install them in their routing and forwarding tables.
5. BGP neighbors advertise their new best routes.
6. The router processes incoming BGP updates, selects new best routes, and installs them in routing and forwarding tables.

Neighbor loss detection can be improved with Bidirectional Forwarding Detection (BFD)^[2], fast neighbor failover^[3] or BGP next-hop tracking. BGP update propagation can be fine-tuned with BGP update timers. The other elements of the BGP convergence process are harder to tune; they depend primarily on the processing power of routers’ CPU, and the underlying packet forwarding hardware.

Some router vendors offer functionality that can be used to pre-install backup paths in BGP tables (BGP best external paths) and forwarding tables (BGP Prefix Independent Convergence^[4]). These features can be used to redirect the traffic to the backup Internet connection even before the BGP convergence process is complete.

Alternatively, you can significantly reduce the CPU load of the Internet edge routers, and improve the BGP convergence time, by reducing the number of BGP prefixes accepted from the upstream ISPs.

Finally, you might need to replace your Internet edge routers with devices that have processing power matching today’s Internet routing table sizes.

Notes

1. BGP Routing Table Analysis Reports
   http://bgp.potaroo.net/
2. Bidirectional Forwarding Detection
   http://wiki.nil.com/Bidirectional_Forwarding_Detection_(BFD)
3. Fast BGP Neighbor Loss Detection
   http://wiki.nil.com/Fast_BGP_neighbor_loss_detection
4. Prefix Independent Convergence – Fixing the FIB Bottleneck
   http://blog.ioshints.info/2012/01/prefix-independent-convergence-pic.html

Get the complete document

Complete case study, including design and deployment guidelines and sample configuration snippets is available to ipSpace.net subscribers. Select the Case studies tab after logging into the webinar management system.

Start now