This page contains information about how to use IP Tunneling to greatly increase the scalability of a virtual server.
IP tunneling (IP encapsulation) is a technique to encapsulate IP datagram within IP datagrams, which allows datagrams destined for one IP address to be wrapped and redirected to another IP address. IP encapsulation is now commonly used in Extranet, Mobile-IP, IP-Multicast, tunneled host or network. Please see the NET-3-HOWTO document for details.
First, let's look at the figure of virtual server via IP tunneling. The most different thing of virtual server via IP tunneling to that of virtual server via NAT is that the load balancer sends requests to real servers through IP tunnel in the former, and the load balancer sends request to real servers via network address translation in the latter.
When a user accesses the service provided by the server cluster, the packet destined for virtual IP address (the IP address for the virtual server) arrives. The load balancer examines the packet's destination address and port. If they are matched for a virtual server service, a real server is chosen from the cluster by a scheduling algorithm, and the connection is added into the hash table which records the established connection. Then, the load balancer encapsulates the packet within an IP datagram and forwards it to the chosen server. When the incoming packet belongs to this connection and the chosen server can be found in the hash table, the packet will be again encapsulated and forwarded to the server. When the server receives the encapsulated packet, it decapsulates the packet and processes the request, finally return the result directly to the user. After the connection terminates or timeouts, the connection record will be removed from the hash table. The workflow is illustrated in the following figure.
Note that real servers can have any real IP address in any network, they can be geographically distributed, but they must support IP encapsulation protocol, and their tunnel devices are all configured as <Virtual IP Address>, like "ifconfig tunl0 <Virtual IP Address>" in Linux. When the encapsulated packet arrives, the real server decapsulates it and finds that the packet is destined for <Virtual IP Address>, it says, "Oh, it is for me, so I do it.", it processes the request and returns the result directly to the user in the end.
First, get a fresh copy of the Linux kernel source of the right version. Second, apply the virtual server patch to the kernel. Third, make sure the following kernel compile options at least are selected.
Kernel Compile Options:
Code maturity level options --->
[*] Prompt for development and/or incomplete code/drivers
Networking options --->
[*] Network firewalls
....
[*] IP: forwarding/gatewaying
....
[*] IP: firewalling
....
[*] IP: masquerading
....
[*] IP: ippfvs(LinuxDirector) masquerading (EXPERIMENTAL)
Virtual server request dispatching technique---
( ) VS-NAT
(X) VS-Tunneling
( ) VS-DRouting
And, you have to choice one scheduling algorithm.
Virtual server scheduling algorithm
(X) WeightedRoundRobin
( ) LeastConnection
( ) WeightedLeastConnection[ ] IP: enabling ippfvs with the local node feature
Fourth, rebuild the kernel. Once you have your kernel properly built, update your system kernel and reboot. At last, cd the ippfvsadm source and type "make install" to install ippfvsadm into your system directory.
Kernel Compile Options:
Code maturity level options --->
[*] Prompt for development and/or incomplete code/drivers
Networking options --->
[*] Network firewalls
....
[*] IP: forwarding/gatewaying
....
[*] IP: firewalling
[*] IP: always defragment (required for masquerading) (NEW)
....
[*] IP: masquerading
....
[*] IP: masquerading virtual server support (EXPERIMENTAL)(NEW)
(12) IP masquerading table size (the Nth power of 2)(NEW)
<M> IPVS: round-robin scheduling(NEW)
<M> IPVS: weighted round-robin scheduling(NEW)
<M> IPVS: weighted least-connection scheduling(NEW)
<M> IPVS: persistent client connection scheduling(NEW)
Once you have your kernel properly built, update your system kernel and reboot. At last, cd the ipvsadm source and type "make install" to install ipvsadm into your system directory.
Let's give an example to see how to use it. The following table illustrates the rules specified in the Linux box with virtual server via IP tunneling. Note that the services running on the real servers must run on the same port as virtual service, so it is not necessary to specify the service port on the real servers.
Protocol | Virtual IP Address | Port | Real IP Address | Weight |
TCP | 202.103.106.5 | 80 | 202.103.107.2 | 1 |
202.103.106.3 | 2 |
All traffic destined for IP address 202.103.106.5 Port 80 is load-balanced over real IP address 202.103.107.2 Port 80 and 202.103.106.3 Port 80.
We can use the following commands to specify the rules in the table above in the system.
ippfvsadm -A -t 202.103.106.5:80 -R 202.103.107.2 -w 1
ippfvsadm -A -t 202.103.106.5:80 -R 202.103.106.3 -w 2
ipvsadm -A -t 202.103.106.5:80 -s wlc (WLC scheduling used for this virtual service)
ipvsadm -a -t 202.103.106.5:80 -R 202.103.107.2 -i -w 1
ipvsadm -a -t 202.103.106.5:80 -R 202.103.106.3 -i -w 2
Here is my configure example for testing virtual server via tunneling. The configuration is as follows. I hope it can give you some clues.
The load balancer (LinuxDirector), kernel 2.0.36
ifconfig eth0 172.26.20.111 netmask 255.255.255.0 broadcast 172.26.20.255 up
route add -net 172.26.20.0 netmask 255.255.255.0 dev eth0
ifconfig eth0:0 172.26.20.110 netmask 255.255.255.255 broadcast 172.26.20.110 up
route add -host 172.26.20.110 dev eth0:0
ippfvsadm -A -t 172.26.20.110:23 -R 172.26.20.112
The real server 1, kernel 2.0.36
ifconfig eth0 172.26.20.112 netmask 255.255.255.0 broadcast 172.26.20.255 up
route add -net 172.26.20.0 netmask 255.255.255.0 dev eth0
ifconfig tunl0 172.26.20.110 netmask 255.255.255.255 broadcast 172.26.20.110 up
route add -host 172.26.20.110 dev tunl0
When I am on other hosts, 'telnet 172.26.20.110' will actually connect the real server 1.
The load balancer (LinuxDirector), kernel 2.2.9
ifconfig eth0 172.26.20.111 netmask 255.255.255.0 broadcast 172.26.20.255 up
route add -net 172.26.20.0 netmask 255.255.255.0 dev eth0
ifconfig eth0:0 172.26.20.110 netmask 255.255.255.255 broadcast 172.26.20.110 up
route add -host 172.26.20.110 dev eth0:0
ipvsadm -A -t 172.26.20.110:23 -s wlc
ipvsadm -a -t 172.26.20.110:23 -R 172.26.20.112 -i
The real server 1, kernel 2.0.36
ifconfig eth0 172.26.20.112 netmask 255.255.255.0 broadcast 172.26.20.255 up
route add -net 172.26.20.0 netmask 255.255.255.0 dev eth0
ifconfig tunl0 172.26.20.110 netmask 255.255.255.255 broadcast 172.26.20.110 up
route add -host 172.26.20.110 dev tunl0
Last updated on June 24, 1999
Created on November 29, 1998