Virtual Server via IP Tunneling

This page contains information about how to use IP Tunneling to greatly increase the scalability of a virtual server.

IP tunneling

IP tunneling (IP encapsulation) is a technique to encapsulate IP datagram within IP datagrams, which allows datagrams destined for one IP address to be wrapped and redirected to another IP address. IP encapsulation is now commonly used in Extranet, Mobile-IP, IP-Multicast, tunneled host or network. Please see the NET-3-HOWTO document for details.

How to use IP tunneling on virtual server

First, let's look at the figure of virtual server via IP tunneling. The most different thing of virtual server via IP tunneling to that of virtual server via NAT is that the load balancer sends requests to real servers through IP tunnel in the former, and the load balancer sends request to real servers via network address translation in the latter.

 

When a user accesses the service provided by the server cluster, the packet destined for virtual IP address (the IP address for the virtual server) arrives. The load balancer examines the packet's destination address and port. If they are matched for a virtual server service, a real server is chosen from the cluster by a scheduling algorithm, and the connection is added into the hash table which record the established connection. Then, the load balancer encapsulates the packet within an IP datagram and forward it to the chosen server. When the incoming packet belongs to this connection and the chosen server can be found in the hash table, the packet will be again encapsulated and forwarded to the server. When the server receives the encapsulated packet, it decapsulate the packet and processes the request, finally return the result directly to the user. After the connection terminates or timeouts, the connection record will be removed from the hash table.

Note that real servers can have any real IP addresses in any network, they can be geographically dispersed, but they must support IP encapsulation protocol, and their tunnel devices are all configured as <Virtual IP Address>, like "ifconfig tunl? <Virtual IP Address>" in Linux. When the encapsulated packet arrives, the real server decapsulates it and finds that the packet is destined for <Virtual IP Address>, it says, "Oh, it is for me, so I do it.", it precesses the request and returns the result directly to the user in the end.

Why use IP tunneling on virtual server

In the virtual server via NAT, incoming packets and replies all need to pass through the load balancer, the load balancer may be a new bottleneck when the number of server nodes increase to 25 or more, because the data throughout may reach the max data through of the network interface. We can see from many Internet services (such as web service) that the incoming packets are always short and reply packets always have large amount of data. In the virtual server via IP tunneling, the load balancer just schedules requests to the different real servers, and the real servers return replies directly to the users. So, the load balancer can handle huge amount of requests, it may schedule over 100 real servers, and it won't be the bottleneck of the system. :-) Thus using IP tunneling will greatly increase the maximum number of server nodes for a load balancer.

The IP tunneling feature can be used to build a very high-performance virtual server, extremely good to build a virtual proxy server, because when the proxy servers get request, it can access the Internet directly to fetch objects and return them directly to the users.

However, all servers must have "IP Tunneling" protocol enabled, I just tested it on Linux IP tunneling. If you make virtual server work on servers running other OS with IP tunneling, please let me know, I will be glad to hear that.

How to build it

First, get a fresh copy of the Linux kernel source of the right version. Second, apply the virtual server patch to the kernel. Third, make sure the following kernel compile options at least are selected.

Kernel Compile Options:

Code maturity level options --->

[*] Prompt for development and/or incomplete code/drivers

Networking options --->

[*] Network firewalls
....
[*] IP: forwarding/gatewaying
....
[*] IP: firewalling
....
[*] IP: masquerading
....
[*] IP: ipportfw masq & virtual server support

And, you have to choice one scheduling algorithm.

Virtual server scheduling algorithm
(X) WeightedRoundRobin
( ) LeastConnection
( ) WeightedLeastConnection

[*] IP: enabling ippfvs using IP tunneling

Fourth, rebuild the kernel. Once you have your kernel properly built, update your system kernel and reboot. At last, cd the ippfvsadm source and type "make install" to install ippfvsadm into your system directory.

How to use it

Let's give an example to see how to use it. The following table illustrates the rules specified in the Linux box with virtual server via IP tunneling. Note that the services running on the real servers must run on the same port as virtual service, so it is not necessary to specify the service port on the real servers.

Protocol Virtual IP Address Port Real IP Address Weight
TCP 202.103.106.5 80 202.103.107.2 1
202.103.106.3 2

All traffic destined for IP address 202.103.106.5 Port 80 is load-balanced over real IP address 202.103.107.2 Port 80 and 202.103.106.3 Port 80.

We can use the following commands to specify the rules in the table above in the system.

ippfvsadm -A -t 202.103.106.5:80 -R 202.103.107.2 -w 1

ippfvsadm -A -t 202.103.106.5:80 -R 202.103.106.3 -w 2

 


Last updated on November 30, 1998

Created on November 29, 1998