Monthly Archives: July 2010

Varnish 2.1.3 released

From varnish-announce@varnish-cache.org :


We are pleased to announce Varnish 2.1.3, a bug fix and feature release in the Varnish 2.1 series.
The most notable changes are:

  • The scalability of critbit, the default hashing method, has been improved
  • A bug in varnishd would in some cases confuse varnishncsa leading to lost or wrong log lines.
  • Some bugs in the handling of Range requests has been fixed. This only matters if you enable Range support.
  • Add «log» command to VCL which will log to the Varnish log.

This is a summary of the changes, please see the changelog for a fuller list.

The release can be downloaded from Sourceforge, as usual: http://sourceforge.net/projects/varnish/

Problem with IIS6 Etag and SetETagFromFileDependencies (.net)

Two problems:

Another way to link Varnish and MaxMind GeoIP

In the tutorial http://varnish-cache.org/wiki/GeoipUsingInlineC, you can see one way to do.
But there is another way, with no need to compile a geoip_plugin.so.

Follow these steps:

  • Go on http://www.maxmind.com/app/c
  • Download http://geolite.maxmind.com/download/geoip/api/c/GeoIP.tar.gz
  • Download the last database http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
  • Install GeoIp:
    • ./configure
    • make
    • make check
    • make install
  • Update your database: GeoIp.dat (/usr/local/share/GeoIP/GeoIP.dat)
  • Edit your /etc/sysconfig/varnish :
    • add :
      -p 'cc_command=exec cc -fpic -shared -Wl,-x -L/usr/include/libmemcached/memcached.h -L/usr/local/lib -lGeoIP -lmemcached -o %o %s' \
    • or if you don’t need memcached:
      -p 'cc_command=exec cc -fpic -shared -Wl,-x -L/usr/local/lib -lGeoIP -o %o %s' \
  • Add the following lines in your vcl
    C{
    
     #include <GeoIP.h>
    
      static GeoIP *gi = NULL;
    
      const char*
      get_country_code(const char* ip)
      {
            const char* country = NULL;
    
            if (gi == NULL)
                    gi = GeoIP_new(GEOIP_STANDARD);
    
            if (ip != NULL)
                    country = GeoIP_country_code_by_addr(gi, ip);
    
            return country ? country : "Unknown";
      }
    
    }C
    
    C{
      VRT_SetHdr(sp, HDR_REQ, "\017X-Country-Code:", (*get_country_code)( VRT_IP_string(sp, VRT_r_client_ip(sp)) ), vrt_magic_string_end);
    }C
    

Now you have the country code in a new header: X-Country-Code

Minify on the fly with ASP.NET and Microsoft IIS

If you are like me, you have always old classic asp applications. May be, these applications are important for your business. So you want optimize these old applications like the others.

One of rules you can find is to minify your CSS and js.

A little research on google, you will find some offline minifier (yui compressor, Microsoft Ajax Minifier)
But nothing if you want an online tool to reduce tasks at minimum when you pass on your production environment.

In the past I have done a tool to resize image by servers on the fly. I used an http httpHandler in c# to do the job.
So in order to minify i’ve taken the same principle.

One http httpHandler in c# with Microsoft Ajax Minifier to do the job.

The normal request: http://example.com/base.CSS
The minify version: http://example.com/base_min_.CSS

// Copyright (c) 2010 OUESTFRANCE-MULTIMEDIA
// All rights reserved.
//
// Author: Vincent ROBERT <v.robert@of2m.fr>
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions
// are met:
// 1. Redistributions of source code must retain the above copyright
//    notice, this list of conditions and the following disclaimer.
// 2. Redistributions in binary form must reproduce the above copyright
//    notice, this list of conditions and the following disclaimer in the
//    documentation and/or other materials provided with the distribution.
//
// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
// ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
// SUCH DAMAGE.

using System;
using System.Collections.Generic;
using System.Text;
using System.Web;
using System.Net;
using System.IO;
using System.Text.RegularExpressions;
using System.IO.Compression;
using System.Configuration;
using System.Security.Cryptography;

public class httpHandlerMinify : IHttpHandler
{
   public bool IsReusable
   {
      get { return false; }
   }

   public void ProcessRequest(HttpContext context)
   {
      try
      {
         string stringToResponse = string.Empty;
         string stringToMinify = string.Empty;
         string contentType = string.Empty;
         Encoding encoding = null;
         TimeSpan settingMaxAge = TimeSpan.Zero;

         string fullFileName = context.Request.PhysicalPath;

         #region debugViaUserAgent
         bool minifyEnable;
         if (context.Request.UserAgent.Contains("NOMINIFY"))
            minifyEnable = false;
         else
            minifyEnable = true;
         #endregion

         #region getAppSettings
         string settingMaxAgeString = ConfigurationManager.AppSettings["MaxAge"];
         if (settingMaxAgeString != string.Empty)
            settingMaxAge = TimeSpan.FromSeconds(Convert.ToDouble(settingMaxAgeString));
         #endregion

         #region loadFile&Minify
         Regex myRegexp = new Regex(@"^(.+)_min_(\.(js|css))$");
         Match myMatch = myRegexp.Match(fullFileName);
         if (myMatch.Success)
         {
            string fileName = myMatch.Groups[1].Value;
            string extension = myMatch.Groups[2].Value;
            fullFileName = fileName + extension;

            openFile(fullFileName, ref stringToMinify, ref encoding);

            Microsoft.Ajax.Utilities.Minifier myMinifier = new Microsoft.Ajax.Utilities.Minifier();

            switch (extension)
            {
               case ".js":
                  if (minifyEnable)
                     stringToResponse = myMinifier.MinifyJavaScript(stringToMinify);
                  else
                     stringToResponse = stringToMinify;
                  contentType = "application/x-javascript";
                  break;
               case ".css":
                  if (minifyEnable)
                     stringToResponse = myMinifier.MinifyStyleSheet(stringToMinify);
                  else
                     stringToResponse = stringToMinify;
                  contentType = "text/css";
                  break;
            }
         }
         else
         {
            openFile(fullFileName, ref stringToResponse, ref encoding);

            if (fullFileName.Contains(".js"))
            {
               contentType = "application/x-javascript";
            }
            else if (fullFileName.Contains(".css"))
            {
               contentType = "text/css";
            }
         }         
         #endregion

         #region sendTheResponse
         if (stringToResponse != string.Empty)
         {
            bool isCompressed = canGZip(context.Request);

            byte[] responseBytes;
            byte[] stringToResponseInBytes = encoding.GetBytes(stringToResponse);

            using (MemoryStream memoryStream = new MemoryStream(stringToResponseInBytes.Length * 2))
            {
               using (Stream writer = isCompressed ?
                          (Stream)(new GZipStream(memoryStream, CompressionMode.Compress)) :
                          memoryStream)
               {
                  writer.Write(stringToResponseInBytes, 0, stringToResponseInBytes.Length);
               }


               responseBytes = memoryStream.ToArray();
            }

            context.Response.AppendHeader("Vary", "Accept-Encoding");
            context.Response.AppendHeader("Content-Length", responseBytes.Length.ToString());
            context.Response.ContentType = contentType;

            if (isCompressed)
               context.Response.AppendHeader("Content-Encoding", "gzip");

            if (minifyEnable)
            {
               context.Response.Cache.SetCacheability(HttpCacheability.Public);

               if (settingMaxAge != TimeSpan.Zero)
                  context.Response.Cache.SetMaxAge(settingMaxAge);

               context.Response.AddFileDependency(fullFileName);
               context.Response.Cache.SetLastModifiedFromFileDependencies();
               context.Response.Cache.SetETagFromFileDependencies();

            }
            else
            {
               context.Response.Cache.SetCacheability(HttpCacheability.NoCache);
            }

            context.Response.Charset = encoding.WebName;
            context.Response.ContentEncoding = encoding;

            context.Response.OutputStream.Write(responseBytes, 0, responseBytes.Length);
            context.Response.Flush();
         }
         else
         {
            context.Response.Cache.SetCacheability(HttpCacheability.Public);

            if (settingMaxAge != TimeSpan.Zero)
               context.Response.Cache.SetMaxAge(settingMaxAge);

            context.Response.ContentType = contentType;
            context.Response.AddFileDependency(fullFileName);
            context.Response.Cache.SetLastModifiedFromFileDependencies();
            context.Response.Cache.SetETagFromFileDependencies();
            context.Response.Write("");
            context.Response.Flush();
         }

         #endregion

      }
      catch (System.IO.FileNotFoundException)
      {
         context.Response.StatusCode = (int)HttpStatusCode.NotFound;
      }
   }

   private static void openFile(string fileName, ref string stringToMinify, ref Encoding encoding)
   {
      encoding = GetFileEncoding(fileName);
      StreamReader sr = new StreamReader(File.OpenRead(fileName), encoding);
      stringToMinify = sr.ReadToEnd();
      sr.Close();

   }

   private static Encoding GetFileEncoding(string srcFile)
   {
      // *** Use Default of Encoding.Default (Ansi CodePage)
      Encoding enc = Encoding.Default;

      // *** Detect byte order mark if any - otherwise assume default
      byte[] buffer = new byte[5];
      FileStream file = new FileStream(srcFile, FileMode.Open, FileAccess.Read);
      file.Read(buffer, 0, 5);
      file.Close();

      if (buffer[0] == 0xef && buffer[1] == 0xbb && buffer[2] == 0xbf)
         enc = Encoding.UTF8;
      else if (buffer[0] == 0xfe && buffer[1] == 0xff)
         enc = Encoding.Unicode;
      else if (buffer[0] == 0 && buffer[1] == 0 && buffer[2] == 0xfe && buffer[3] == 0xff)
         enc = Encoding.UTF32;
      else if (buffer[0] == 0x2b && buffer[1] == 0x2f && buffer[2] == 0x76)
         enc = Encoding.UTF7;

      return enc;
   }


   private static bool canGZip(HttpRequest request)
   {
      string acceptEncoding = request.Headers["Accept-Encoding"];
      if (!string.IsNullOrEmpty(acceptEncoding) && acceptEncoding.Contains("gzip"))
         return true;
      return false;
   }
}

You can download the code here.

In order to use the httpHandler, you need to do the following things:

  • Put the dlls (AjaxMin.dll and httpHandlerMinify.dll) in the GAC (C:\windows\Assembly)
  • Modify your web.config in C:\Windows\Microsoft.NET\Framework\v2.0.50727\CONFIG :
    <add assembly="httpHandlerMinify, Version=1.0.0.0, Culture=neutral, PublicKeyToken=04966276bde0d064" />
  • Modify web.config in your website folder :
    <appSettings>      
    	<add key="MaxAge" value="3600"/>
    </appSettings>

    and

    <httpHandlers>
             <add verb="*" path="*.js,*.css," type="httpHandlerMinify,httpHandlerMinify, Version=1.0.0.0, Culture=neutral, PublicKeyToken=04966276bde0d064" validate="false"/>
    </httpHandlers>
  • Modify your IIS website configuration :

You can follow this tutorial: http://msdn.microsoft.com/en-us/library/ms972953.aspx

How to obtain Time Taken with varnishncsa ?



There is a way to add the time taken to serve the request in the varnishncsa logs.

You must use the version 2.1.2 of Varnish.
Then apply the following patch http://varnish-cache.org/ticket/712 , you can download here.


In order to use the new option, launch the varnishncsa with the command line:

varnishncsa -L "%h %l %u %t \"%{VarnishR}i\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D"

%D refer to “The time taken to serve the request, in microseconds”.


Here a list of options:

  • %H, Protocol version
  • %{Host}i
  • %{Referer}i
  • %U, URL path
  • %q, the query string
  • %U%q, URL path and query string
  • %{User-agent}i
  • %{X-Forwarded-For}i
  • %b, Bytes
  • %h (host name / IP adress)
  • %m, Request method
  • %s, Status
  • %t, Date and time
  • %u, Remote user
  • %D, The time taken to serve the request, in microseconds
  • %{X-Cache-Age}i Age of served object

I don’t know if the patch is official.
Be careful on production, i’ve some segfault:

kernel: varnishncsa[29466]: segfault at 0000000000000000 rip 000000368a2797e0 rsp 00007fff0348dbf8 error 4

Update May 2013:
With Varnish 3, you can get the status (hit or miss) and the time taken (time to first bytes) easily with varnishncsa:

varnishncsa -F "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" %{Varnish:hitmiss}x %{Varnish:time_firstbyte}x"

Varnish:hitmiss : Whether the request was a cache hit or miss. Pipe and pass are considered misses.
Varnish:time_firstbyte : Time to the first byte from the backend arrived

Check here for further documentation

Updating Anti-Scraping Captcha



Today, we spotted some IP located in the same area that have won the captcha challenge within a very short time.

So we decided to change the system of captcha.

We stop with the phpcaptcha system, now we use the reCaptcha of google.
The system seems to be stronger.

Varnish VCL to patch IIS6 Compression



In IIS6, static compression is happening on a separate thread. So on receiving a request, first response is uncompressed and IIS used to start a separate thread to compress the file and keep it in compressed files cache.

Problem, Varnish keeps the uncompressed version of the file.

So I’ve added few line to my vcl_fetch:

#patch compression IIS6 (first demand is not compressed)
 if (req.http.Accept-Encoding) {
    if (!(obj.http.Content-Encoding)) {
       set obj.ttl = 0s;
    }
 }

Be careful to remove Accept-Encoding for binaries files, otherwise images can’t be cached:

if (req.http.Accept-Encoding) {
        if (req.url ~ "(?i)\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)(\?.*)?$") {
            # No point in compressing these
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            remove req.http.Accept-Encoding;
        }
    }