Load Balancing

Pools

To configure Pools:

  • Navigate to Services > Load Balancer
  • Click the Pools tab
  • Click Pluse Add to add a new pool
  • Configure the pool options as explained below:

Name A name for the Pool. The name is how the pool is referenced when configuring the Virtual Server that will use this pool. This name must adhere to the same limits as an alias or interface name. Letters and numbers only, the only allowed separator is an underscore.

Note

This name cannot be the same as an existing alias.

Mode Select Load Balance to balance load between all servers in the pool, or Manual Failover to always use the servers in the Enabled list, and they can be manually moved between an enabled and disabled state.

Description A optional longer description for the Pool.

Port This is the port the servers are listening oninternally. This can be different from the external port, which is defined later in the Virtual Server configuration. An alias may also be used to define multipleports, however, if an alias is used it must use the same port alias here and in the Virtual Server configuration.

Retry This defines the number of times a server will be contacted by the monitor before being declared down.

Monitor This defines the type of monitor to use, which is how the load balancer determines if the servers are up and usable. Selecting TCP will make the balancer connect to the port previously defined in Port, and if it cannot connect to that port, the server is considered down. Choosing ICMP will instead monitor the defined servers by sending an ICMP ping, and will mark them down if they do not respond. There are many more types of monitors, and they can be customized. They are covered in more detail later in the chapter.

Server IP Address This is where the internal IP addresses of the servers in the pool are listed. Enter them one at a time, clicking Add to pool afterwards.

Current Pool Members This field shows the list of servers in this pool. A server can be removed from the pool by clicking on its IP address and then clicking Remove. There are two lists in this section, Pool Disabled, and Enabled (default). The servers in the Enabled (default) list are active and used, servers in the Pool Disabled list are never used. The Pool Disabled list is primarily used with Manual Failover mode. Servers can be moved between the lists by selecting them and clicking >> or >>.

  • Click Save

If automatic failover is required, create a second poolto be used as a Fall Back Pool, containing the backup set of server IP addresses.

Virtual Servers

To configure a Virtual Server to handle client connections:

  • Navigate to Services > Load Balancer
  • Click the Virtual Servers tab
  • Click pluse Add to add a new Virtual Server
  • Configure the Virtual Server options as explained below:

Name A name for the Virtual Server. This is for reference, but must also adhere to the same limits as an alias or interface name. Letters and numbers only, the only allowed separator is an underscore. No spaces or slashes.

Description An optional longer description for theVirtual Server. This is for reference purposes only, and does not have any formatting limits.

IP Address This is where IP addresses are entered for use by the Virtual Server. This is usually the WAN IP address or a Virtual IP address on WAN. It must be astatic IP address. A CARP VIP may also be used for a high availability setup. For more information on high availability and CARP VIPs, refer to High Availability. AnIP Alias VIP may be used, or a Proxy ARP VIP (TCP mode only). Furthermore, an Alias may also be used here to specify multiple IP addresses upon which this Virtual Server may accept connections.

Note

In TCP mode, the IP addresses specified here are not bound at the OS level, meaning that relayd as a daemon is not bound and listening on these ports directly.

Port This is the port upon which the Virtual Server will accept connections. It can be different from theport used by the pool servers internally. An alias can be used to define multiple ports, however, if the same port alias must be used here and in the Pool configuration.

Virtual Server Pool This is where the previously configured pool is selected. The connections to the IPAddress and Port defined on this screen will be directed to the IP addresses and ports configured in thepool.

Fall Back Pool This is the alternate pool that clients are directed to if all the servers in the primary pool are down. If there is no alternate server, leave this set to None, though the result will be inaccessibility if all the servers in the pool are down. If nothing else, to avoid having the server be down entirely, setup a simple web server to return abasic maintenance page for any request and use it as the fall back pool.

Relay Protocol The Relay Protocol can be either TCP or DNS, depending on what this relay will be doing.

– In TCP mode, relayd acts like an enhanced port forward, directing connections as though they were hitting a traditional NAT rule. Servers will see the original source IP address of the client, it does not act as a proxy.

– In DNS mode, relayd acts as a DNS proxy. It will balance the load over multiple DNS servers, but the original client IP address is lost. Pool servers will see the firewall as the source of the DNS query. Keep this in mind when setting up views or source-based query restrictions on DNS servers involved in load balancing.

  • Click Submit
  • Click Apply Changes

Warning

If all Virtual Server Pool members and Fall Back Pool members are down, relayd will act as thoughthe Load Balancer is not handling connections for the Virtual Server IP address and port. If the IP address and port used are also used by another service or NAT rule, it could be accidentally exposed to clients.

Monitors

There are five basic pre-defined Monitor types: ICMP, TCP, HTTP, HTTPS, and SMTP. Additional custom types may be added to better detect specific types of failures.

Pre-defined Monitors

The pre-defined monitors are included in the default configuration and are:

ICMP Sends an ICMP echo request to the target server and expects an ICMP echo reply.

TCP Attempts to open a TCP port connection to the target IP address and port. If the port can be opened (3-way TCP handshake) then it succeeds, if it connection is refused or timed out, it fails.

HTTP & HTTPS Attempts to open a connection to the server and request the URL / using HTTP or HTTPS, whichever is selected. If a 200 response code is returned, it is OK. Otherwise, it is considered a failure.

SMTP Opens a connection to the defined port and sends the string EHLO nosuchhost. If the server replies with any message starting with 250-, it is considered OKOther responses are considered a failure.

Creating Custom Monitors

The the included monitors are not sufficient for the needs of a site, or they need tweaking, then custom monitors may be created. Most monitor types have their own specific settings that can be customized as needed.

To create a new monitor: Navigate to Services > Load Balancer Click the Monitors tab Click pluse to add a new Monitor Configure the Monitor options as explained below:

Name A name for the Monitor. This is for referencebut must also adhere to the same limits as an alias or interface name. Letters and numbers only, the only allowed separator is an underscore. No spaces or slashes.

Description An optional longer description for theMonitor. This is for reference purposes only, and does not have any formatting limits.

The remaining options vary based on the selected Type.

ICMP & TCP No extra options. Any custom monitor using these types will behave identically to the pre-defined monitor of the same name.

HTTP & HTTPS These behave identically to each otherthe only difference is whether or not encryption is used to talk to the target server. These each have three options to control the behavior of the monitor:

Path The Path defines the path section of the URL sent to the server. If the site contains mostly dynamiccontent, or the base URL does a redirect, it is best to set this to a full path to a static piece of content, such as an image, that is unlikely to move or change.

Host If the server runs multiple virtual hosts, this field defines which hostname is sent with the request so that the expected response can be received.

HTTP Code This defines the response expected from the server, given the request to the Host/Path. Most commonly this would be set to 200 OK, but if the server uses another return code that would be expected as a healthy response to this query, choose it here. If the return code is unknown, inspect the server logs to find what codes are returned to the client for each request.

Send/Expect This type of monitor opens a connection to the defined port and sends a string and expects the specified response. The most common example is the SMTP monitor discussed previously. The options are:

Send String The string sent to the server after a connection is made to its port.

Expect String If the response from the server doesnot start with this string, then it is considered down.

  • Click Save

Settings

In addition to the per-pool or per-server options, there are also a few global options that control the behavior of relayd. These settings are under Services > Load Balancer on the Settings tab:

Timeout The global timeout in milliseconds for checks. Leave blank to use the default value of 1000 ms (1 second). If a loaded server pool takes longer to respond to requests, increase this timeout.

Interval The interval in seconds at which the member ofa pool will be checked. Leave blank to use the default interval of 10 seconds. To check the servers more (or less) frequently, adjust the timing accordingly.

Prefork Number of processes used by relayd for handling inbound connections to relays. This option is only active for relays using DNS mode. It does not have any effect on TCP mode since that uses a redirect, not a relay. Leave blank to use the default value of 5 processes. If the server is busy, increase this amount to accommodate the load.

Firewall rules

The last step in configuring Load Balancing is to configure firewall rules to allow traffic to the pool.

For TCP mode, the firewall rules must permit traffic tothe internal private IP addresses of the servers, the same as with NAT rules, as well as the port they are listening on internally. Create an alias for the servers in the pool to make the process easier, and create a single firewall rule on the interface where the traffic destined to the pool will be initiated (usually WAN) allowing the appropriate source (usually any) to a destination of the alias created for the pool.

A specific example of this is provided in Configuring firewall rules. For more information on firewall rules, refer to Firewall.

For DNS mode, firewall rules must allow traffic directly to the Virtual Server IP address and port, not the pool servers.

Sticky connections

There is one additional configuration option available for server load balancing, under System > Advanced,on the Miscellaneous tab. Under Load Balancing,called Use sticky connections. Checking this box will attempt to send clients with an active connection to thepool server to the same server for any subsequent connections.

Once the client closes all active connections, and the closed state times out, the sticky connection is lost. This may be desirable for some web load balancing configurations where client requests must only go to a single server, for session or other reasons. This isn’t perfect, as if theclient’s web browser closes all TCP connections to the server after loading a page and sits there for 10 minutes or more before loading the next page, the next page may be served from a different server. Generally this isn’t an issue as most web browsers won’t immediately close a connection, and the state exists long enough to not make it a problem, but if the site is strictly reliant on a specific client never getting a different server in the pool regardless of how long thebrowser sits there inactive, look for a different loadbalancing solution. There is a box under the option to control the Source Tracking Timeout which can allow the knowledge of the client/server relationship to persist longer.

Warning

Sticky is generally unreliable for this purpose and can also have other unintended side effects.Full featured proxy packages such as HAProxy have much better mechanisms and options for maintaining client/server relationships.

There are four areas of configuration for the server load balancer:

  • Pools define collections of servers to be used, which port they use, and the monitoring method.
  • Virtual Servers define the IP address and port for accepting user connections, and the appropriate poolto direct the incoming traffic destined to that IP address and port.
  • Monitors are used to create custom monitoring methodsto determine if pool servers are working and usable.
  • The Settings tab contains global options that alter how the load balancer operates.

In a typical example, there is a Virtual Server to accept user connections, and it contains several servers in a Pool. The Pool utilizes a Monitor for each server to determine if it is capable of accepting user connections.

A Virtual Server can have a regular and a Fall Back Pool to use if all members of the regular Virtual Server Pool are down. This can be leveraged to present a maintenance or outage page, for example.

Web Server Load Balancing Example Configuration

This section shows how to configure the Load Balancer from start to finish for load balanced environment with two web servers.

Example network environment

Figure Server Load Balancing Example Network shows the example environment configured in this section. It consists of a single firewall, using its WAN IP address forthe pool, with two web servers on a DMZ segment.

put image

Configuring pool

To configure the pool:

  • Navigate to Services > Load Balancer
  • Click the Pools tab
  • Click pluse Add to create a new pool
  • Configure the pool as shown in Figure Pool Configuration, which uses the following settings:

Name WebServers

Mode Load Balance

Description Web server Pool

Port 80

Retry 5

Pool Members Add both web servers (10.6.0.11 and 10.6.0.12) using an HTTP Monitor

  • Click Save

Configuring virtual server

  • Click the Virtual Servers tab
  • Click pluse Add to add a new virtual server
  • Configure the Virtual Server as shown in Figure Virtual Server Configuration, which uses the following settings:

Name WebVirtualServer

Description Web Server

IP Address The firewall’s WAN IP address, 198.51.100.6

Port 80

Virtual Server Pool WebServers

Fall Back Pool None

  • Click Submit

put image

  • Click Apply Changes

Warning

In this example, if both of the pool servers are down, the VirtualServer is inaccessible. The firewall will act as if no Virtual Server is configured. If something on the firewall is bound to port 80, clients will reach that instead. This includes the built-in Web GUI redirect for port 80, so that should be disabled under System > Advanced on the Admin Access tab.

Configuring firewall rules

Firewall rules must be configured to allow access to the servers in the pool. The rules must allow the traffic to the internal IP addresses and port being used,and no rules are necessary for the outside IP Address and Port used in the virtual server configuration.

Create an alias containing all the servers in the pool, so access can be allowedwith a single firewall rule.

  • Navigate to Firewall > Aliases
  • Click pluse Add to add an alias.
  • Use the following settings:

Name www_servers

Type Hosts

Hosts The IP addresses of both web servers: 10.6.0.11 and 10.6.0.12

  • Click Save
  • Click Apply Changes

put image

Next, create a firewall rule using that alias: Navigate to Firewall > Rules**Change to the tab for the interface where connections will enter (e.g. WAN) Click **upper arrow Add to start a new rule at the top of the list * Use the following.

settings:

Interface WAN

Protocol TCP

Source any

Destination Type Single Host or Alias

Destination Address www_servers

Destination Port Range HTTP

Description Allow to Web Server

  • Click Save
  • Click Apply Changes

Adding Firewall Rule for Web Servers shows a snippet of the firewall rule added for this configuration. The options not shown are left at their defaults.

put image

Viewing load balancer status

Now that the load balancer is configured, to view its status, browse to Status> Load Balancer and click the Virtual Servers tab. This page displays the status of the server as a whole, typically listed as either Active or Down The Pools tab shows an individual status for each member of a Pool (as shown in Figure Pool Status). The row for a server is green if it is online, and red if the server is offline.

Additionally, each server in the pool has a checkbox next to it. Servers that are checked are active in the pool, and unchecked servers are disabled in the pool,the same as moving them between the enabled and disabled list on the pool editing page. To disable a server: Uncheck it, then click Save.

put image

If the web server service is stopped on one of the servers, or if the server is removed from the network entirely if using ICMP monitors, the status updates to Offline and the server is removed from the pool.

Verifying load balancing

To verify load balancing, curl is the best option to ensure the web browser cache and persistent connections do not affect the results of testing. curl is available for every OS imaginable and can be downloaded from the curl website. To use it, simply run:

curl http://mysite

In that command, replace 198.51.100.6 with either the IP address or hostname of the site. This must be tested from outside the network (e.g. from a remote network or client on WAN). The following illustrates an example of testing with curl from the WAN side:

# curl http://198.51.100.6 This is server www2 - 10.6.0.12 # curl http://198.51.100.6 This is server www1 - 10.6.0.11

When initially testing load balancing, configure each server to return a page specifying its hostname, IP address, or both, so it is made obvious which server is responding to the request. If sticky connections is not enabled, a different server will respond to each request.