linuxmonk.ch

I recently had the task to setup and test a new Linux Internet gateway host as a replacement to an existing router. The setup is classical with some individual ports such as HTTP being forwarded with DNAT to some backend systems.

Redundant router setup requiring policy routing

The new router, from now on I will call it router #2, should be tested and put into operation without downtime. Obviously this means that I had to run the two routers in parallel for a while. The backend systems however, only know one default gateway. Accessing a service through a forwarded port on router #2 resulted in a timeout, as the backend system sent the replies to the wrong gateway where they were dropped.

Fortunately iptables and iproute2 came to my rescue. They enable you to implement policy routing on Linux. This means that the routing decision is not (only) made based on the destination address of a packet as in regular routing, but additional rules are evaluated. In my case: Every connection opened through router #2 has to be replied via router #2.

Using iptables/iproute2 for this task this means: Incoming packets with the source MAC address from router #2 are marked with help of the iptables ‘mark’ extension. The iptables ‘connmark’ extension will then help to associate outgoing packets to the previously marked connection. Based on the mark of the outgoing packet a custom routing policy will set the default gateway to router #2. Easy, eh?

Configuration
Now I’ll show some commands how this can be accomplished. The following commands assume that the iptables rule set is still empty and are for demonstration purpose only. Likely they have to be adjusted slightly in a real configuration.

First the routing policy will be setup:

1. Define a custom routing table. There exist some default tables, so the custom entry shouldn’t overlap with those. For better understanding I will call it ‘router2’:

   # echo "200 router2" >> /etc/iproute2/rt_tables

2. Add a rule to define the condition which packets should lookup the routing in the previously created table ‘router2’:
   &nbps;
   # ip rule add fwmark 0x2 lookup router2

   This means that IP packets with the mark ‘2’ will be routed according to the table ‘router2’.

3. Set the default gateway in the ‘router2’ table to the IP address of router #2 (e.g. 10.0.0.2):

   # ip route add default via 10.0.0.2 table router2

4. To make sure the routing cache is rebuilt, it need to be flushed after changes:
   # ip route flush cache

Afterwards I had to make sure that the involved connections coming from router #2 are marked appropriately (above the mark ‘2’ was used). The ‘mangle’ table is a part of the Linux iptables packet filter and meant for modifying network packets. This is the place where the packet markings will be set.

1. The first iptables rule will match all packets belonging to a new connection coming from router #2 and sets the previously defined mark ‘2’:

   # iptables --table mangle --append INPUT \
--protocol tcp --dport 80 \
--match state --state NEW \
--match mac --mac-source 52:54:00:c2:a5:43 \
! --source 10.0.0.0/24 \
--jump MARK --set-mark 0x2

   The packets being marked are restricted to meet the following requirements:

   • being sent by the network adapter of router #2 (--mac-source)
   • don’t originate in the local network (! --source)
   • target destination port 80 (--dport: example for the HTTP port being forwarded by router #2)
   • belong to a new connection (--state NEW)

   Of course additional (or less) extensions can be used to filter the packets according to individual requirements.

2. Next, the incoming packets are given to the ‘connmark’ extension which will do the connection tracking of the marked connections:

   # iptables --table mangle --append INPUT \
--jump CONNMARK --save-mark

3. The packets which can be associated with an existing connection are also marked accordingly:

   # iptables --table mangle --append INPUT \
--match state --state ESTABLISHED,RELATED \
--jump CONNMARK --restore-mark

4. All the previous rules where required that the outgoing packets can finally be marked too:

   # iptables --table mangle --append OUTPUT \
--jump CONNMARK --restore-mark

Debugging
The following commands and iptables rules should help when setting up and/or debugging policy routing with marked packets:

• List policy of routing table ‘table2’:

  # ip route show table router2

• List defined routing tables:

  # cat /etc/iproute2/rt_tables

• Log marked incoming/outgoing packets to syslog:

  # iptables -A INPUT -m mark --mark 0x2 -j LOG
# iptables -A OUTPUT -m mark --mark 0x2 -j LOG