Tuesday, September 17, 2013

Using Varnish Proxy Cache with Amazon Web Service ELB Elastic Load Balancer


using-varnish-proxy-cache-with-amazon-web-services-elastic-load-balancer-elb


Update 19-Feb-2014 ! Elastic Load Balancing Announces Cross-Zone Load Balancing
Maybe this new option makes unnecessary my workaround. Anyone can confirm?


The problem
When putting a Varnish cache in front of an AWS EC2 Elastic Load Balancer weird things happen like: Not getting any traffic to your instance or getting traffic to just one of your instances (in case of Multi Availability Zone (AZ) deployment).

Why?
This has to do with how the ELB is designed and how Varnish is designed. Is not a flaw. Let's call it: Incompatibility.
When you deploy a Elastic Load Balancer into EC2 you access it through a CNAME DNS address. When you deploy an ELB in front of multiple instances in multiple Availability Zones that CNAME is not a DNS address, is many.

Example:
$ dig www.netflix.com

; <<>> DiG 9.8.1-P1 <<>> www.netflix.com

;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64502
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:

;www.netflix.com. IN A

;; ANSWER SECTION:

www.netflix.com. 300 IN CNAME dualstack.wwwservice--frontend-san-756424683.us-east-1.elb.amazonaws.com.
dualstack.wwwservice--frontend-san-756424683.us-east-1.elb.amazonaws.com. 60 IN A 184.73.248.59
dualstack.wwwservice--frontend-san-756424683.us-east-1.elb.amazonaws.com. 60 IN A 107.22.235.237
dualstack.wwwservice--frontend-san-756424683.us-east-1.elb.amazonaws.com. 60 IN A 184.73.252.179


As you can see, the answer for this CNAME DNS resolution for Netflix's ELB are 3 different IP addresses. Is up to the application (usually your Internet Web Browser) to decide which to use. Different clients will chose different IPs (they are not always sorted the same way) and this will balance the traffic among different AZs.
The bottom line is that your ELB in real life are multiple instances in multiple AZs and the CNAME mechanism is the method used to balance them.

But Varnish behaves different
And when you specify a CNAME as a Varnish backend server (the destination server where Varnish requests will be send to) it will translate that into only one IP. Despite the amount of IP addresses associated with that CNAME. It will only chose one and use that one for all its activity. Therefore Varnish and AWS ELB are not compatible. (Would you like to suggest a change?)

The Solution
Put a NGINX web server between Varnish and the ELB, acting as a load balancer. I know, not elegant. but works and once is in place no maintenance is needed and the process overhead for the Varnish server is minimum.

Setup
- Varnish server listening on TCP port 80 and configured to send all its requests to 127.0.0.1:8080
- NGINX server listening on TCP port 127.0.0.1:8080 and sending all its requests to our EC2 ELB.

Basic configuration (using AWS EC2 AMI Linux)

yum update
reboot

yum install varnish
yum install nginx

chkconfig varnish on
chkconfig nginx on

Varnish

vim /etc/sysconfig/varnish

Locate the line:
VARNISH_LISTEN_PORT=6081
and change if for
VARNISH_LISTEN_PORT=80

vim /etc/varnish/default.vcl

Locate the backend default configuration and change port from 80 to 8080
backend default {
  .host = "127.0.0.1";
  .port = "8080";
}


NGINX

vim /etc/nginx/nginx.conf

Get rid of the default configuration file and use this example:
worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    keepalive_timeout  65;

    server_tokens off;

server {
    listen       localhost:8080;

    location / {
     ### Insert below your ELB DNS Name leaving the semicolon at the end of the line
     proxy_pass  http://<<<<Insert-here-your-ELB-DNS-Name>>>>;
     proxy_set_header Host $http_host;

   }
}

}


Restart
service varnish restart
service nginx restart

And voila! Comments and improvement are welcome.


Thanks to
Jordi and Àlex for your help!


Update 19-Feb-2014 ! Elastic Load Balancing Announces Cross-Zone Load Balancing
Maybe this new option makes unnecessary my workaround. Anyone can confirm?


4 comments:

  1. FYI

    http://gc-taylor.com/blog/2011/11/10/nginx-aws-elb-name-resolution-resolvers/

    Nginx does not play nicely with ELB UNLESS you dynamically state the ELB DNS in a variable that needs to be checked within an internal ttl.

    As per page above:

    "If you are running nginx as a proxy in front of An Amazon Web Services Elastic Load Balancer (ELB), it is not safe to merely define an upstream using the hostname of ELB and call it a day. By default, nginx will only do name resolution at startup time, caching the resolved IP address infinitely."

    Since ELB's change their backend IP address, Nginx will end up having a problem as it does not check the resolution of the DNS record that relates to the ELB DNS name EVER apart from at startup, by default.

    you have to put a variable into the proxy_pass variable typically, I prefer the hostname, but the request url can be used as well.

    See http://forum.nginx.org/read.php?2,238835,238899#msg-238899 as well as the above url for examples.

    Steve

    ReplyDelete
  2. Hi, Is it possible to use an http apache server in stead of an nginx ? Thanks!

    ReplyDelete
    Replies
    1. Hi Luis,
      I really don't know. You need a proxy module for Apache and some research to be sure that it handles well the ELB'S CNAMEs.
      I strongly suggest NGINX. It is a tiny config and works like a charm.

      Delete
  3. These innovations use correspondence foundations that are both worldwide and dependably up, in this manner empowering 24-hour activity and offbeat just as synchronous connections among people, gatherings, and associations. prywatnoscwsieci.pl

    ReplyDelete