Tag Archives: varnish

ModPageSpeed in front of Varnish

10 Replies

At work, we are using Varnish, a very good reverse proxy, since many years.
Varnish does a lot of stuff for us. One of the major is the ability to cache responses of our quite slow php backends. Getting a page in 16ms is ever better than 300ms :)

Since my venue to the London Velocity Conf, I am quite obsessed to test Google ModPagespeed.
The product is not focused on the frontend response time but more on the user experience.
By providing many filters, it can optimized web page applying rules like: CSS minifying, optimizing images, defer JavaScript and many others. Nothing we can’t do ourselves but it can save times of dev teams.

I have done my first test of ModPageSpeed few weeks ago. Results seems very good but one things hurts me. ModPageSpeed proceed in two stage optimization: the first call of the page, ModPageSpeed get the page from PHP and returns the response without any optimization to the client. And in parallel it launches a thread to optimize the page for the next client.
As it don’t optimize the first call, it forces a cache-control “no-cache” on the response.
The problem is that no page will go in the cache of Varnish with a no-cache directive.
One last thing, ModPageSpeed sends different optimizations depending of the User-Agent.

So, one side we have optimized the HTML of the response, increasing user experience, but on the other side we have put more load on our backend.

Thinking about it, I asked myself if we could put ModPageSpeed in front of Varnish. If yes, we could have an optimized HTML and less load on the backend.

The response is yes, we can put ModPageSpeed in front of Varnish !

We just need to install Apache and configure it with “ProxyPass”.

This is all the modules we need:

LoadModule authz_host_module /usr/lib64/httpd/modules/mod_authz_host.so
LoadModule deflate_module /usr/lib64/httpd/modules/mod_deflate.so
LoadModule log_config_module /usr/lib64/httpd/modules/mod_log_config.so
LoadModule setenvif_module /usr/lib64/httpd/modules/mod_setenvif.so
LoadModule proxy_module /usr/lib64/httpd/modules/mod_proxy.so
LoadModule proxy_http_module /usr/lib64/httpd/modules/mod_proxy_http.so
LoadModule status_module /usr/lib64/httpd/modules/mod_status.so
LoadModule vhost_alias_module /usr/lib64/httpd/modules/mod_vhost_alias.so

All others modules can be safely disabled.

Since ModPagespeed uses threads, think to start your Apache in worker mode: 

HTTPD=/usr/sbin/httpd.worker

in 

/etc/sysconfig/httpd

If you have more than one vhost, you need to use at least the 1.1.23.2-2191 of ModPagespeed to support multi-vhosts.

A default setup could be:

DocumentRoot "/var/www/htdocs"

<Directory />
  Order allow,deny
  Allow from all
</Directory>

Include conf.d/pagespeed.conf

NameVirtualHost *:80

<VirtualHost *:80>
  #Default VHost : ModPagespeed disabled
  ModPagespeed Off
  CustomLog /var/log/httpd/default_access.log combined
  Errorlog /var/log/httpd/default_error.log
</VirtualHost>

<VirtualHost *:80>
  #mywebsite.com VHost : ModPagespeed enabled
  ServerName www.mywebsite.com
  ServerAlias s1.mystaticwebsite.com
  ServerAlias s2.mystaticwebsite.com
  ModPagespeed On
  ModPagespeedEnableFilters lazyload_images,collapse_whitespace,combine_javascript,defer_javascript
  ModPagespeedMapOriginDomain http://localhost:8000 http://www.mywebsite.com
  ModPagespeedMapOriginDomain http://localhost:8000 http://mystaticwebsite.com
  ModPagespeedShardDomain mystaticwebsite.com s1.mystaticwebsite.com,s2.mystaticwebsite.com
  CustomLog /var/log/httpd/www.mywebsite.com_access.log combined
  Errorlog /var/log/httpd/www.mywebsite.com_error.log
</VirtualHost>

ProxyRequests Off
ProxyPreserveHost On
ProxyPass / http://127.0.0.1:8000/

This configuration is fully operational but not yet tested in production.

Is there someone who has already tested this architecture in production?

Varnish 3.0.2 released

Leave a reply

Summary of changes from 3.0.1 to 3.0.2

  • A crasher bug when requests were queued and the backend sent a response with Vary has been fixed.
  • A crash when a too large synthetic response was produced has been fixed.
  • The ban lurker now properly sleeps the 1 second it is supposed to.
  • Varnish now releases disk space properly if no -s argument is provided, and the default cache size is now 100MB instead of 50% of the available disk space.

Download here.

CTRL F5 to force the update of Varnish cache

1 Reply

Add the following code to your vcl, so you can use CTRL F5 to force a refresh:

acl CTRLF5 {
   "xxx.xxx.xxx.xxx";
}

sub vcl_hit {

  if (client.ip ~ CTRLF5) {
    if (req.http.pragma ~ "no-cache" || req.http.Cache-Control ~ "no-cache") 
    {
      set obj.ttl = 0s; 
      return(pass);
    }
    else { return(deliver); }
  }
  else { return(deliver); }
}

Update May 2013:
With Varnish 3, no need to use obj.ttl = 0s, you just need to call purge method.

acl CTRLF5 {
   "xxx.xxx.xxx.xxx";
}

sub vcl_hit {

  if (client.ip ~ CTRLF5) {
    if (req.http.pragma ~ "no-cache" || req.http.Cache-Control ~ "no-cache") 
    {
      purge; 
      return(pass);
    }
    else { return(deliver); }
  }
  else { return(deliver); }
}

Few corrections for wordpress.vcl in the wordpress plugin for Varnish

1 Reply

I’ve just migrated this blog on a dedicated server.
In order to use Varnish 3.0.1, i have done few ajustements on the wordpress.vcl contained in the wordpress plugin for Varnish

backend default {
  .host = "127.0.0.1";
  .port = "8080";
}

acl purge {
  "localhost";
}

sub vcl_recv {
  if (req.request == "PURGE") {
    if(!client.ip ~ purge) {
      error 405 "Not allowed.";
    }

    return (lookup);
  }

  if (req.request != "GET" &&
      req.request != "HEAD" &&
      req.request != "PUT" &&
      req.request != "POST" &&
      req.request != "TRACE" &&
      req.request != "OPTIONS" &&
      req.request != "DELETE") {
    return (pipe);
  }

  if (req.request != "GET" && req.request != "HEAD") {
    return (pass);
  }

  if (req.url ~ "wp-(login|admin)" || req.url ~ "preview=true") {
    return (pass);
  }

  remove req.http.cookie;
  return (lookup);
}

sub vcl_fetch {
  if (req.url ~ "wp-(login|admin)" || req.url ~ "preview=true") {
    return (hit_for_pass);
  }

  set beresp.ttl = 24h;
  return (deliver);
}

sub vcl_hit {
  if (req.request == "PURGE") {
     purge;
     error 200 "Purged.";
  }

  return(deliver);
}

sub vcl_miss {
  if (req.request == "PURGE") {
    purge;
    error 200 "Purged.";
  }

  return (fetch);
}

Sample of string concatenation in Varnish 2.1.5

Leave a reply

In Varnish 2.1.5, to concatenate, separate your variables by a space character :

set req.url = regsub(req.http.host, "(?:www\.)?(.*)\.test\.com", "/\1" ) regsub(req.url, "^/foo(.*)$", "\1");

If ( req.http.host == “www.blabla.test.com” ) and ( req.url == “/foo/index.php?a=1″ ) Then req.url = “/blabla/index.php?a=1″

Since Varnish 3.0, the string concatenation operator is +.
Example: 

set req.url = regsub(req.http.host, "(?:www\.)?(.*)\.test\.com", "/\1" ) + regsub(req.url, "^/foo(.*)$", "\1");

Varnish 3.0.1 released

Leave a reply

Summary of changes from 3.0.0 to 3.0.1

  • Objects with grace and keep set were mistakenly seen as candidates for the shortlived storage, but would not be cleaned up quickly, something that manifested as if there was a memory leak. This is now fixed.
  • When multiple clients were waiting for an object, all clients would be woken up when an object became available, leading to stuck threads. This has now been fixed.
  • A bug in how XML entities were handled with ESI has been fixed.
  • The documentation has seen numerous updates.
  • varnishncsa is now more stable and has support for showing arbitrary request and response fields.

Download here.

Varnish 3.0.0 released

3 Replies

Summary of changes from 2.1.5 to 3.0.0

  • Module support through VMODs.
  • Compression and uncompression support, including stiching together compressed ESI fragments.
  • Preliminary streaming support, both on miss and on pass.
  • Much improved documentation.
  • Better default values for parameters.
  • Varnishncsa now has custom log format support.
  • Varnishlog, varnishncsa and varnishhist can now filter out records that match multiple expressions.

Download here.

Need to purge Varnish of all websites containing a string ?

Leave a reply

With varnish 2.1.4, try this:

if you want purge this regexp: ^(www\.)?(.*)xxx

you must do : varnishadm -T localhost:6082 purge "req.http.host ~ \"^(www\\\.)?(.*)xxx\""

Be careful to put good backslash sequence, in order to avoid: Syntax Error: Invalid backslash sequence

Varnish 2.1.5 released

Leave a reply

Summary of changes from 2.1.4 to 2.1.5

  • Two bugs relating to Content-Length and possible duplication of Content-Length headers have been resolved.
  • Support for bourne-like “here”-documents in the command line interface, allowing <<__EOF__ and similar schemes.
  • Fixed an issue with re-using connections after Chunked-Encoding.
  • Fix a bug that would inflate the “lost header” count and could cause problems during heavy traffic over a single connection, typically seen by load testing.
  • Use the time of cache-insertion for “If-Modified-Since” requests if a “Last-Modified” header isn’t provided by the backend.
  • Merge multi-line Vary and Cache-Control headers from clients, which Google Chromium seem to split up.
  • Various build fixes and documentation improvements
  • Various bug fixes.

Download here.

In response “to the cookie monster”

4 Replies

Today I read the article Perbu on the “cookie monster”. http://www.varnish-software.com/blog/cookie-monster

I do not agree with “If the client sends you a cookie Varnish will not issue a cached page but instead fetch a new page from the backend servers”.

This is more complicated. Varnish does not care if the client (the browser) send a cookie or not. But Varnish take care of the backend response. If the backend sends a response with a cookie, the page can’t be put in the Varnish Cache.

I’m very sure of that, we have a lot of cookie for analytics purpose in our page, they are put by javascript on the client side. For each request of the browser, you can see (using LiveHttpHeader) that cookie are sent to Varnish and yet, Varnish continue to hit in the cache.