ModPageSpeed in front of Varnish

At work, we are using Varnish, a very good reverse proxy, since many years.
Varnish does a lot of stuff for us. One of the major is the ability to cache responses of our quite slow php backends. Getting a page in 16ms is ever better than 300ms :)

Since my venue to the London Velocity Conf, I am quite obsessed to test Google ModPagespeed.
The product is not focused on the frontend response time but more on the user experience.
By providing many filters, it can optimized web page applying rules like: CSS minifying, optimizing images, defer JavaScript and many others. Nothing we can’t do ourselves but it can save times of dev teams.

I have done my first test of ModPageSpeed few weeks ago. Results seems very good but one things hurts me. ModPageSpeed proceed in two stage optimization: the first call of the page, ModPageSpeed get the page from PHP and returns the response without any optimization to the client. And in parallel it launches a thread to optimize the page for the next client.
As it don’t optimize the first call, it forces a cache-control “no-cache” on the response.
The problem is that no page will go in the cache of Varnish with a no-cache directive.
One last thing, ModPageSpeed sends different optimizations depending of the User-Agent.

So, one side we have optimized the HTML of the response, increasing user experience, but on the other side we have put more load on our backend.

Thinking about it, I asked myself if we could put ModPageSpeed in front of Varnish. If yes, we could have an optimized HTML and less load on the backend.

The response is yes, we can put ModPageSpeed in front of Varnish !

We just need to install Apache and configure it with “ProxyPass”.

This is all the modules we need:

LoadModule authz_host_module /usr/lib64/httpd/modules/mod_authz_host.so
LoadModule deflate_module /usr/lib64/httpd/modules/mod_deflate.so
LoadModule log_config_module /usr/lib64/httpd/modules/mod_log_config.so
LoadModule setenvif_module /usr/lib64/httpd/modules/mod_setenvif.so
LoadModule proxy_module /usr/lib64/httpd/modules/mod_proxy.so
LoadModule proxy_http_module /usr/lib64/httpd/modules/mod_proxy_http.so
LoadModule status_module /usr/lib64/httpd/modules/mod_status.so
LoadModule vhost_alias_module /usr/lib64/httpd/modules/mod_vhost_alias.so

All others modules can be safely disabled.

Since ModPagespeed uses threads, think to start your Apache in worker mode:

HTTPD=/usr/sbin/httpd.worker

in

/etc/sysconfig/httpd

If you have more than one vhost, you need to use at least the 1.1.23.2-2191 of ModPagespeed to support multi-vhosts.

A default setup could be:

DocumentRoot "/var/www/htdocs"

<Directory />
  Order allow,deny
  Allow from all
</Directory>

Include conf.d/pagespeed.conf

NameVirtualHost *:80

<VirtualHost *:80>
  #Default VHost : ModPagespeed disabled
  ModPagespeed Off
  CustomLog /var/log/httpd/default_access.log combined
  Errorlog /var/log/httpd/default_error.log
</VirtualHost>

<VirtualHost *:80>
  #mywebsite.com VHost : ModPagespeed enabled
  ServerName www.mywebsite.com
  ServerAlias s1.mystaticwebsite.com
  ServerAlias s2.mystaticwebsite.com
  ModPagespeed On
  ModPagespeedEnableFilters lazyload_images,collapse_whitespace,combine_javascript,defer_javascript
  ModPagespeedMapOriginDomain http://localhost:8000 http://www.mywebsite.com
  ModPagespeedMapOriginDomain http://localhost:8000 http://mystaticwebsite.com
  ModPagespeedShardDomain mystaticwebsite.com s1.mystaticwebsite.com,s2.mystaticwebsite.com
  CustomLog /var/log/httpd/www.mywebsite.com_access.log combined
  Errorlog /var/log/httpd/www.mywebsite.com_error.log
</VirtualHost>

ProxyRequests Off
ProxyPreserveHost On
ProxyPass / http://127.0.0.1:8000/

This configuration is fully operational but not yet tested in production.

Is there someone who has already tested this architecture in production?

When the optimization increases our response time due to a bug in the Firefox Navigation Timing API

This morning, I was very embarrassed when I looked at our Google Analytics Site Speed statistics.

One of our sites has risen from 4s to 8s response time!

What have we done that could explain this?

After the reading of a very good post of Aaron Peters Why loading third party scripts async is not good enough, i thought many scripts can be loaded after the “onload” event.

Yesterday, we have implemented a new version of one of our websites, with the loading of “Google Plus One” and “Criteo” after Onload.

For those who are interested by the criteo script, take at look at: the js.

Loading of Criteo and “Google Plus One” is done by the same way of Aaron Peters:

<script>
    window.___gcfg = {
        lang: 'fr'
    };
    (function(w, d, s) {
      function go(){
        var js, fjs = d.getElementsByTagName(s)[0], load = function(url, id) {
         if (d.getElementById(id)) {return;}
         js = d.createElement(s); js.src = url; js.id = id;
         fjs.parentNode.insertBefore(js, fjs);
        };
        load('/js/criteo.js', 'criteojs');
        load('https://apis.google.com/js/plusone.js', 'gplus1js');
      }
      if (w.addEventListener) { w.addEventListener("load", go, false); }
      else if (w.attachEvent) { w.attachEvent("onload",go); }
    }(window, document, 'script'));
</script>

So, why the response time is increasing ? It should be better but certainly not worse.

When we look carefully to the response time for web browser, we see that the effect of optimization is not the same for all.
On the one hand, we have IE and Chrome, that have better time after the optimization, on the other hand we have Firefox that is worse.

January 03, 2012

  • IE = 3.78s
  • Firefox = 5.31s
  • Chrome = 3.55s

January 04, 2012

  • IE = 3.33s (-12%)
  • Firefox = 17.18s (+223%)
  • Chrome = 2.99s (-16%)

Now let’s look more closely at the statistics for Firefox.
January 03, 2012

  • Firefox 7.0.1 : 2.54s
  • Firefox 8.0.0 : 6.00s
  • Firefox 8.0.1 : 5.87s
  • Firefox 9.0.1 : 4.04s


January 04, 2012

  • Firefox 7.0.1 : 21.73s (+755%)
  • Firefox 8.0.0 : 21.53s (+258%)
  • Firefox 8.0.1 : 18.69s (+218%)
  • Firefox 9.0.1 : 2.89s (-28%)

So what’s the problem with Firefox 7 & 8 ?

A little bit of searching on google gave me a answer: the problem is due to a bug on the Navigation Time API on firefox 7 & 8.

The bug is referenced on bugzilla n°691547.
Yan can take at look on the mailing list: http://lists.w3.org/Archives/Public/public-web-perf/2011Nov/0039.html

So to summarize, optimization, consisting of some scripts to load after the onload of the page, lowers the loading time, triggering event “onload” earlier.
This is true for all browsers.
But a bug in Firefox (<9) broke the statistics of loading time.
The problem is that google think our website has become slower even though it has become faster.

So what should we do?


Edit of January 06, 2012:

We try this workaround:

<script>
    window.___gcfg = {
        lang: 'fr'
    };
    (function(w, d, s) {
      function go(){
        var js, fjs = d.getElementsByTagName(s)[0], load = function(url, id) {
         if (d.getElementById(id)) {return;}
         js = d.createElement(s); js.src = url; js.id = id;
         fjs.parentNode.insertBefore(js, fjs);
        };
        load('/js/criteo.js', 'criteojs');
        load('https://apis.google.com/js/plusone.js', 'gplus1js');
      }
      if (jQuery.browser.mozilla && parseInt(jQuery.browser.version, 10) < 9) {
        go();
      } else {
        if (w.addEventListener) { w.addEventListener("load", go, false); }
        else if (w.attachEvent) { w.attachEvent("onload",go); }
      }
    }(window, document, 'script'));
</script>

We will see in few days if the statistics are better.


Edit of January 13, 2012:

As you see, the workaround is good.

Comparison between Dec 31,2011 – Jan 3,2012 (before onload) and Jan 9,2012 – Jan 12,2012 (after onload):

Firefox 7 / 8 are still slower when scripts are loaded after onload. Fortunately, the number of visit of these browsers falls sharply in favor of Firefox 9.

Apache Segmentation fault with Newrelic PHP Agent 2.6.5.44 and PHP 5.1.6

After upgrading Newrelic PHP agent from 2.5.5.29 to 2.6.5.44, we have detected some Apache Segmentation fault on our servers.

Newrelic never officially supported PHP 5.1, but it worked so far.

The only workaround I found is to downgrade Newrelic at 2.5.5.29. You can download the rpm here :http://yum.newrelic.com/php_agent/archive/2.6.5.41/newrelic-php5-2.5.5.29-1.x86_64.rpm


Edit of January 17, 2012:

Newrelic just released version 2.6.5.57 of the PHP agent. We have updated and everything works fine. Segfault disappeared.

Varnish 3.0.2 released

Summary of changes from 3.0.1 to 3.0.2

  • A crasher bug when requests were queued and the backend sent a response with Vary has been fixed.
  • A crash when a too large synthetic response was produced has been fixed.
  • The ban lurker now properly sleeps the 1 second it is supposed to.
  • Varnish now releases disk space properly if no -s argument is provided, and the default cache size is now 100MB instead of 50% of the available disk space.

Download here.

SQL Server 2008 : Failed to notify ‘xxxx’ via email.

If you have difficulties to send email by SQL Server Agent:

  • Failed to notify ‘xxxx’ via email

and Datebase mail “Send Test E-Mail” is ok, try this:

  • Verify SQL Server Agent Error Logs, you will certainly see
    • An attempt was made to send an email when no email session has been established

    .

  • If you see this message, then:
    • right click on SQL Server Agent, Properties, Alert System: “Enable mail profile” must be checked. (not checked by default)

Can’t run vbscript with 32 bits ActiveX on Windows 2008 64 bits ?

On the 64-bit Windows operating system, there are 64-bit and 32-bit versions of cscript.exe and wscript.exe.

Typically, the script hosts reside in the following directories:

  • 64-bit script hosts:
    • C:\WINDOWS\system32\cscript.exe
    • C:\WINDOWS\system32\wscript.exe
  • 32-bit script hosts:
    • C:\WINDOWS\SysWOW64\cscript.exe
    • C:\WINDOWS\SysWOW64\wscript.exe

The 32-bit script hosts can create and use only 32-bit COM components, and the 64-bit script hosts can create and use only 64-bit COM components.

By default, on 64-bit Windows operating systems, VBScript files are associated with the 64-bit script hosts.

Same things when you want registering DLL:

  • C:WINDOWS\SysWOW64\regsvr32.exe to register 32-bits DLL
  • C:WINDOWS\System32\regsvr32.exe to register 64-bits DLL

Same things again when you want registering a dotnet assembly:

  • C:WINDOWS\Microsoft.Net\Framework\v2.0.50727\regasm.exe to use a .net framework 2.0 assembly in a 32-bit Vbscript (lauched by a 32-bit cscript)
  • C:WINDOWS\Microsoft.Net\Framework64\v2.0.50727\regasm.exe to use a .net framework 2.0 assembly in a 64-bit Vbscript (lauched by a 64-bit cscript)

CTRL F5 to force the update of Varnish cache

Add the following code to your vcl, so you can use CTRL F5 to force a refresh:

acl CTRLF5 {
   "xxx.xxx.xxx.xxx";
}

sub vcl_hit {

  if (client.ip ~ CTRLF5) {
    if (req.http.pragma ~ "no-cache" || req.http.Cache-Control ~ "no-cache") 
    {
      set obj.ttl = 0s; 
      return(pass);
    }
    else { return(deliver); }
  }
  else { return(deliver); }
}

Update May 2013:
With Varnish 3, no need to use obj.ttl = 0s, you just need to call purge method.

acl CTRLF5 {
   "xxx.xxx.xxx.xxx";
}

sub vcl_hit {

  if (client.ip ~ CTRLF5) {
    if (req.http.pragma ~ "no-cache" || req.http.Cache-Control ~ "no-cache") 
    {
      purge; 
      return(pass);
    }
    else { return(deliver); }
  }
  else { return(deliver); }
}

Few corrections for wordpress.vcl in the wordpress plugin for Varnish

I’ve just migrated this blog on a dedicated server.
In order to use Varnish 3.0.1, i have done few ajustements on the wordpress.vcl contained in the wordpress plugin for Varnish

backend default {
  .host = "127.0.0.1";
  .port = "8080";
}

acl purge {
  "localhost";
}

sub vcl_recv {
  if (req.request == "PURGE") {
    if(!client.ip ~ purge) {
      error 405 "Not allowed.";
    }

    return (lookup);
  }

  if (req.request != "GET" &&
      req.request != "HEAD" &&
      req.request != "PUT" &&
      req.request != "POST" &&
      req.request != "TRACE" &&
      req.request != "OPTIONS" &&
      req.request != "DELETE") {
    return (pipe);
  }

  if (req.request != "GET" && req.request != "HEAD") {
    return (pass);
  }

  if (req.url ~ "wp-(login|admin)" || req.url ~ "preview=true") {
    return (pass);
  }

  remove req.http.cookie;
  return (lookup);
}

sub vcl_fetch {
  if (req.url ~ "wp-(login|admin)" || req.url ~ "preview=true") {
    return (hit_for_pass);
  }

  set beresp.ttl = 24h;
  return (deliver);
}

sub vcl_hit {
  if (req.request == "PURGE") {
     purge;
     error 200 "Purged.";
  }

  return(deliver);
}

sub vcl_miss {
  if (req.request == "PURGE") {
    purge;
    error 200 "Purged.";
  }

  return (fetch);
}

Sample of string concatenation in Varnish 2.1.5

In Varnish 2.1.5, to concatenate, separate your variables by a space character :

set req.url = regsub(req.http.host, "(?:www\.)?(.*)\.test\.com", "/\1" ) regsub(req.url, "^/foo(.*)$", "\1");

If ( req.http.host == “www.blabla.test.com” ) and ( req.url == “/foo/index.php?a=1″ ) Then req.url = “/blabla/index.php?a=1″

Since Varnish 3.0, the string concatenation operator is +.
Example:

set req.url = regsub(req.http.host, "(?:www\.)?(.*)\.test\.com", "/\1" ) + regsub(req.url, "^/foo(.*)$", "\1");