BGP: Three Uplinks. Ingoing And Outgoing Traffic Balancing

We have an autonomous system (AS), a few blocks of provider-independent addresses (PI), three external channels (uplinks) with different speed. And we use BGP (board-gateway protocol) to exchange with route tables.
We should configure the load balancing of incoming and outgoing traffic roughly evenly between all channels (uplinks). And we should remember in the mind about given speed each of the channels. We should also set up an automatic routing to live channel(s) in case of some channel(s) is broken, – and return to its former state in case of channel(s) recovery.

As a BGP router we’re using a server running Linux with Quagga installed.  Quagga is a software to do BGP work.

I’ll not discuss all config options  - the basics on how to configure BGP can explore the Internet. I’ll touch on only the important questions about the topic.

So. We write our AS number and id of our router:

router bgp 12345
bgp router-id 1.1.1.1

Then rewrite the network that are connected directly to us, and which we are eager to announce to the world:

network 1.1.1.0 mask 255.255.255.0
network 2.2.2.0 mask 255.255.255.0
network 3.3.3.0 mask 255.255.255.0
network 4.4.4.0 mask 255.255.254.0

Now we describe our neighbors, with whom we will do BGP exchanges:

neighbor 111.111.111.111 remote-as AS1
neighbor 111.111.111.111 description UPLINK_1
neighbor 111.111.111.111 update-source 111.111.111.112
neighbor 111.111.111.111 weight 3000
neighbor 111.111.111.111 route-map prepend_uplink1 out
neighbor 111.111.111.111 prefix-list plup1in in

where:

update-source 111.111.111.112 – can be necessary. that we provide, from which IP address our server should connecte BGP neighbor to. If you have provider-independed adress blocks,  the router can begin connection to the neighbors with the IP from your network;
neighbor 111.111.111.111 weight 3000 – Weight. If you have received from several neighbors routes to the same direction, then in the routing table gets one route, which came from the neighbor with the largest number weight;
neighbor 111.111.111.111 route-map prepend_uplink1 out – assign a route-map to this neighbor. This we use for balancing the incoming traffic;
neighbor 111.111.111.111 prefix-list plup1in in – prefix-list for this neighbor. We will use it for balancing outbound traffic.

Then paint two more neighbors by analogy.

neighbor 222.222.222.222 remote-as AS2
neighbor 222.222.222.222 description UPLINK_2
neighbor 222.222.222.222 update-source 222.222.222.223
neighbor 222.222.222.222 weight 4000
neighbor 222.222.222.222 route-map prepend_uplink2 out
neighbor 222.222.222.222 prefix-list plup2in in
!
neighbor 333.333.333.333 remote-as AS3
neighbor 333.333.333.333 description UPLINK_3
neighbor 333.333.333.333 update-source 333.333.333.334
neighbor 333.333.333.333 weight 2000
neighbor 333.333.333.333 route-map prepend_uplink3 out
neighbor 333.333.333.333 prefix-list plup3in in
!

We now describe our prefix-lists for each neighbor.

For a neighbor number “2″ we prohibits default route and allow reception of only routes with a netmask less than or equal to 19 (<= 19). Other routes we will not take from it.

ip prefix-list plup2in seq 5 deny 0.0.0.0 / 0
ip prefix-list plup2in seq 15 permit 0.0.0.0 / 0 le 19
ip prefix-list plup2in seq 25 deny any

For a neighbor number “1″ we forbid only receive the default route from it:

ip prefix-list plup1in seq 5 deny 0.0.0.0 / 0
ip prefix-list plup1in seq 25 permit any

For a neighbor number “3″, we also simply prohibits default route:

ip prefix-list plup3in seq 5 deny 0.0.0.0 / 0
ip prefix-list plup3in seq 25 permit any

That is, from one of the uplinks (neighbors) we only accept network the mask is equal to or less than 19. But from two anothers we accept all other routes. Default route not accept from anyone – we will put it into config later.

Now let’s look at route-maps. To properly work with route-maps we need to write a couple more prefix lists for each neighbor. These prefix-lists we need to regulate the incoming traffic. The point: for each neighbor we must describe what the network we will announce with bigger prepends, and with – which smaller prepends. For every neighbor we make bigger prepends to one networks and smaller prepends to anothers, and so on. Thus it turns out that from one neighbor to us will be coming traffic is mainly going to some of our network, and from the another – on our other networks. So by the following prefix lists we describe our network.

ip prefix-list plup1 permit 4.4.4.0/23
ip prefix-list plup1 deny any
ip prefix-list plup1p permit 2.2.2.0/24
ip prefix-list plup1p permit 1.1.1.0/24
ip prefix-list plup1p permit 3.3.3.0/24
ip prefix-list plup1p deny any

Here we have created two prefix-list. One gets our network 4.4.4.0/23, another one – the rest of our networks. Just paint the prefix-lists for the remaining neighbors. Only a network will vary.

ip prefix-list plup2 permit 2.2.2.0/24
ip prefix-list plup2 deny any
ip prefix-list plup2p permit 4.4.4.0/23
ip prefix-list plup2p permit 1.1.1.0/24
ip prefix-list plup2p permit 3.3.3.0/24
ip prefix-list plup2p deny any
!
ip prefix-list plup3 permit 1.1.1.0/24
ip prefix-list plup3 permit 3.3.3.0/24
ip prefix-list plup3 deny any
ip prefix-list plup3p permit 4.4.4.0/23
ip prefix-list plup3p permit 2.2.2.0/24
ip prefix-list plup3p deny any

The time has come to describe all of our route-maps for each neighbor. The idea is that for each uplink we have a route-map, that specify which network to announce with a prepend (and how much) and which – advertise directly (without any prepends).

route-map prepend_uplink1 permit 10
match ip address prefix-list plup1
!
route-map prepend_uplink1 permit 20
match ip address prefix-list plup1p
set as-path prepend 12345 12345

By these route-maps we indicated that our network of prefix list plup1 (and this is only the network 4.4.4.0/23) through uplink UPLINK_1 (111.111.111.111) will be announced without any change in the length of the route. Networks of the prefix list plup1p (the three the rest of our network) through the same uplink (UPLINK_1) we will announce with a prepend and artificially lengthened the route to the networks of the world through that uplink with the two hop (two AS). With other uplinks everything are exactly the same, – only the route we lengthened to our another networks.

route-map prepend_uplink2 permit 10
match ip address prefix-list plup2
!
route-map prepend_uplink2 permit 20
match ip address prefix-list plup2p
set as-path prepend 12345 12345
!
route-map prepend_uplink3 permit 10
match ip address prefix-list plup3
!
route-map prepend_uplink3 permit 20
match ip address prefix-list plup3p
set as-path prepend 12345 12345 12345 12345
!

Now what about default route. Even if you receive full view from your uplinks it would be better to have default route in config.

So, login into Zebra with vtysh, and specify default routes. In our case we will have even three default routes which points to our uplinks but with different distances:

ip route 0.0.0.0 / 0 111.111.111.111 15
ip route 0.0.0.0 / 0 222.222.222.222 25
ip route 0.0.0.0 / 0 333.333.333.333 35

In other words, we direct our default route to all three of our uplinks. In the routing table gets that route, whose distance is less than the others. If a route is broken off, the traffic will go to one of the remaining whose distance is now smaller – and so on.

What we got in the end …

UPLINK_1 (111.111.111.111). From it we get all routes except the default. Announce network 4.4.4.0 directly, the others – with lengthening of the route (with prepends). Priority is 3000 (average).
UPLINK_2 (222.222.222.222). From it we get only those routes that point to the network with a mask less than or equal 19. Announce network 2.2.2.0 directly, the others – with prepends to the route. Priority 4000 – the highest.
UPLINK_3 (333.333.333.333). From it we get all routes except the default route. Announce networks 1, 2 and 3 directly, the others  (network 4.4.4.0) – with prepends to the route. Priority 2000 – the very least.

What happened. Since UPLINK_3 has the smallest priority, then through him we do not go (although we get from him just in case all the routes). As with UPLINK_2 we get only part of the route (network mask <= 19), but his priority is highest – that is just on these networks with a mask less than or equal to 19 we go through this uplink. But all the other networks we will go through UPLINK_1 – because with it we get all the routes and his priority is medium. The default route we just ran through UPLINK_1.

Thus, some traffic goes through UPLINK_1, part – through UPLINK_2. Now if you fall off one of them – the traffic will go through the default route, i.e. through one of them is (according to the specified distance) – who will still be alive by then. If the disappearance of the two – the game turns UPLINK_3, as the last remaining and having the longest distance. If you fall off UPLINK_3 – traffic will go through one of the first two uplinks. However, through UPLINK_3 our outbound traffic, and so is only in the event of an accident – he is the essentially as a reserve.

With outbound traffic sorted out. Now what about ingoing traffic.

When everything is OK the traffic will go like this: at the 4-th network traffic would come mainly from the 1 st uplink, 2 nd – the second, 1 second and 3rd – Uplink number 3 . Most likely, the traffic in each uplink will be slip to each of your networks, but primarily on every uplink traffic will go to the networks – which at this uplink is announced without prepends. Traffic on the other networks on this uplink will be the overwhelming minority or unavailable (as will be mainly going through another uplink, where the network been announced without prepends).

In general, the point is made. Now the main thing. What kind of network, how and where to advertise, what routes from where the – all recognizable mainly by experimental methods.

In the above-described technology has to evenly distribute the load of both incoming and outgoing traffic between all uplinks. It provides full viability of the network if at least one of the uplinks is alive. In the case of recovery “fallen” uplink, including them in the work is done automatically and the whole system comes to normal which was before the fall.

So, in case of accidents, human intervention in the work of BGP and routing is absolutely not required – the router is an excellent job himself.