Today is a big day ! This is the first test of our blocking scraping system on production.
We have put it on two of our three varnish.

For a first test we don’t want to block scrapers but just log them.

So we wrote a light version of the script.

# Copyright (c) 2010 OUESTFRANCE-MULTIMEDIA
# All rights reserved.
#
# Author: Vincent ROBERT <v.robert@of2m.fr>
# Thanks to: Tony FOUCHARD <t.fouchard@of2m.fr>
# Thanks to: Cyrille MAHIEUX <c.mahieux@of2m.fr>
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.

C{
  #define HOSTNAME_MAX_SIZE 200
  #include <string.h>
  #include <libmemcached/memcached.h>
  #include <syslog.h>
  #include <netdb.h>
  #include <sys/types.h>
  #include <sys/socket.h>
}C

sub check_ip {
  set bereq.http.whitelisted = "0";
  if (bereq.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|js|css|swf|xml|flv)(\?(.)*)?$") {
     set bereq.http.whitelisted = "1";
  }
  else {
     if (bereq.http.host == "internalwebSite") {
       set bereq.http.whitelisted = "1";
     }
     elsif ((bereq.http.X-Forwarded-For == "IPaddress1") || (bereq.http.X-Forwarded-For == "IPaddress2")) {
       set bereq.http.whitelisted = "1";
     }
  }

  if (bereq.http.whitelisted=="0") {
    C{

      void ip_to_hostname(char *ipaddr, char *hostname)
      {
        struct addrinfo * result;
        int successfull=0;
        int error;

        error = getaddrinfo(ipaddr, NULL, NULL, &result);
        if (error==0) {
           error = getnameinfo(result->ai_addr, result->ai_addrlen, hostname, HOSTNAME_MAX_SIZE, NULL, 0, 0);
           if (error==0) {
             successfull=1;
           }
        }

        freeaddrinfo(result);

        if (!successfull){
          memcpy(hostname, ipaddr, HOSTNAME_MAX_SIZE*sizeof(char));
          hostname[HOSTNAME_MAX_SIZE-1]=0;
        }
      }

      memcached_server_st *servers = NULL;
      memcached_st *memc;
      memcached_return rc;
      char hostname[HOSTNAME_MAX_SIZE]="";

      char key[500]= "varnish_";
      strcat(key, VRT_GetHdr(sp, HDR_REQ, "\005host:"));
      strcat(key, "_");
      strcat(key, VRT_IP_string(sp, VRT_r_client_ip(sp)));

      memc= memcached_create(NULL);
      servers= memcached_server_list_append(servers, "memcacheServer", 11211, &rc);
      rc= memcached_server_push(memc, servers);

      if (rc == MEMCACHED_SUCCESS) {

        uint64_t intval;
        rc= memcached_increment(memc, key, strlen(key), (uint64_t)1, &intval);

        if (rc != MEMCACHED_SUCCESS)
          rc= memcached_set(memc, key, strlen(key), "1", 1, (time_t)60, (uint32_t)0);
        else
          if (intval>30) {
            ip_to_hostname(VRT_IP_string(sp, VRT_r_client_ip(sp)), hostname);
            VRT_SetHdr(sp, HDR_BEREQ, "\013X-Scraping:", "1", vrt_magic_string_end);
            VRT_SetHdr(sp, HDR_BEREQ, "\013X-Hostname:", hostname, vrt_magic_string_end);
            VRT_SetHdr(sp, HDR_BEREQ, "\013X-HostOrig:", VRT_GetHdr(sp, HDR_REQ, "\005host:"), vrt_magic_string_end);
            if (intval<300)
              rc= memcached_set(memc, key, strlen(key), "500", 3, (time_t)600, (uint32_t)0);
          }
      }
      memcached_free(memc);
      memcached_server_list_free(servers);
    }C
    if (bereq.http.X-Scraping=="1") {
      if (!(bereq.http.X-Hostname ~ "(\.(yahoo\.(net|com))|(exabot\.com)|(googlebot\.com)|(search\.msn\.com))$")) {
        C{
          syslog(LOG_INFO, "Scraping detected from %s %s on %s with %s",VRT_IP_string(sp, VRT_r_client_ip(sp)), VRT_GetHdr(sp, HDR_BEREQ, "\013X-Hostname:"), VRT_GetHdr(sp, HDR_REQ, "\005host:"), VRT_GetHdr(sp, HDR_REQ, "\013User-Agent:"));
        }C
      }
    }
  }
}

sub vcl_miss {
  call check_ip;
  fetch;
}

sub vcl_pipe {
  call check_ip;
  pipe;
}

sub vcl_pass {
  call check_ip;
  pass;
}

The load is about 600 requests/s on each Varnish server.
We put just one memcached server.

The result are:

• No CPU overhead on Varnish or too small to be viewed
• No memory leak spotted.
• Memcached server is almost idle: about 60 requests/s and 80 KBytes used !
• Turn off the memcached server don’t affect Varnish and user experience
• Good news, the system show us a lot of scrappers and many spiders
• Bad news, we have detected one or two proxy as scrappers, but they are not.

Here, a piece of our logs :

Jun  9 10:26:40 ouest-proxy03 varnishd[4015]: Scraping detected from 94.23.240.216 ns209194.ovh.net on www.ouest-france.fr with posh
Jun  9 10:29:28 ouest-proxy03 varnishd[4015]: Scraping detected from 94.23.198.123 ns207043.ovh.net on www.ouest-france.fr with Synthesio Crawler release MonaLisa (contact at synthesio dot fr)
Jun  9 10:29:55 ouest-proxy03 varnishd[4015]: Scraping detected from 69.191.249.201 injr-spdrproxy1.bloomberg.com on www.ouest-france.fr with BLP_bbot/0.1
Jun  9 10:29:59 ouest-proxy03 varnishd[4015]: Scraping detected from 89.122.29.79 adsl89-122-29-79.romtelecom.net on www.ouestfrance-emploi.com with Java/1.6.0_04

---

Using phpMemCacheAdmin, you can see memcache server is fine:

---

Here the mk-query-digest profiling of memcache server TCPdump:

 ./mk-query-digest --type memcached --group-by res --limit 10 dump_zym.log

# 69.3s user time, 440ms system time, 83.70M rss, 171.79M vsz
# Current date: Tue May 18 12:17:09 2010
# Files: dump_zym.log
# Overall: 70.68k total, 3 unique, 67.20 QPS, 0.01x concurrency __________
#                    total     min     max     avg     95%  stddev  median
# Exec time             9s    20us    22ms   122us   224us   233us    98us
# Time range        2010-05-18 11:45:40.061323 to 2010-05-18 12:03:11.908228
# bytes             17.63k       0       3    0.26    0.99    0.43       0
#  74% (53k)  Memc_incr
#  25% (18k)  Memc_miss
#  25% (18k)  Memc_set

# Profile
# Rank Query ID Response time    Calls R/Call   Item
# ==== ======== ================ ===== ======== =========
#    1 0x           4.9508 57.3% 34673   0.0001
#    2 0x           2.4583 28.5% 17995   0.0001 NOT_FOUND
#    3 0x           1.2310 14.2% 18015   0.0001 STORED

---

Here the top of one of the Varnish:

top - 16:38:02 up 57 days,  5:09,  3 users,  load average: 0.36, 0.54, 0.54
Tasks: 104 total,   1 running, 103 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.0%us,  1.1%sy,  0.1%ni, 93.2%id,  0.0%wa,  0.3%hi,  0.3%si,  0.0%st
Mem:  33030932k total, 29536176k used,  3494756k free,   781740k buffers
Swap: 11727408k total,     3792k used, 11723616k free,   727416k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2016 varnish   20   0 35.1g  26g  81m S   38 84.2 122:50.26 varnishd
24769 root      20   0 98.8m  80m  80m S    1  0.3   0:03.26 varnishncsa

---

Here the varnishstat of one of our Varnish:

0+07:03:59                                                                                                                                                                      ouest-proxy02.sdv.fr
Hitrate ratio:       10      100     1000
Hitrate avg:     0.7814   0.7972   0.8006

     3693144       103.00       145.18 Client connections accepted
    13991870       556.00       550.02 Client requests received
    11372520       469.00       447.05 Cache hits
       14376         1.00         0.57 Cache hits for pass
     2546655        85.00       100.11 Cache misses
     2622997        87.00       103.11 Backend conn. success
          41         0.00         0.00 Backend conn. not attempted
         507         0.00         0.02 Backend conn. failures
          96         0.00         0.00 Backend conn. reuses
         996         0.00         0.04 Backend conn. was closed
        1093         0.00         0.04 Backend conn. recycles
          36         0.00         0.00 Fetch head
     2338109        71.00        91.91 Fetch with Length
      226850        15.00         8.92 Fetch chunked
         932         0.00         0.04 Fetch wanted close
        1540          .            .   N struct sess_mem
         691          .            .   N struct sess
     2151242          .            .   N struct object
      877521          .            .   N struct objecthead
          56          .            .   N struct vbe_conn
         341          .            .   N struct bereq
         144          .            .   N worker threads
        1474         0.00         0.06 N worker threads created
           0         0.00         0.00 N queued work requests
       19144         0.00         0.75 N overflowed work requests
          29          .            .   N backends
         549          .            .   N expired objects
      395693          .            .   N LRU nuked objects
      345902          .            .   N LRU moved objects
          23         0.00         0.00 HTTP header overflows
    10527830       435.00       413.85 Objects sent with write
          12         0.00         0.00 Objects overflowing workspace
     3693079       113.00       145.17 Total Sessions
    13991789       557.00       550.01 Total Requests
       57104         2.00         2.24 Total pipe
       19567         0.00         0.77 Total pass
     2565891        90.00       100.86 Total fetch
  5115095581    210266.25    201072.98 Total header bytes
 71922504943   2823420.83   2827253.62 Total body bytes
      779458        22.00        30.64 Session Closed
         407         0.00         0.02 Session Pipeline
         317         0.00         0.01 Session Read Ahead
    13466511       535.00       529.36 Session Linger
    12094595       508.00       475.44 Session herd
   680468640     26550.16     26749.03 SHM records
    40799330      1602.01      1603.81 SHM writes
       52523         2.00         2.06 SHM flushes due to overflow
       64353         1.00         2.53 SHM MTX contention
         292         0.00         0.01 SHM cycles through buffer
     5476474       264.00       215.28 SMA allocator requests
     4256875          .            .   SMA outstanding allocations
 32212244810          .            .   SMA outstanding bytes
 63522967274          .            .   SMA bytes allocated
 31310722464          .            .   SMA bytes free
        1494         0.00         0.06 SMS allocator requests
      776444          .            .   SMS bytes allocated
      776444          .            .   SMS bytes freed
     2566011        86.00       100.87 Backend requests made
           4         0.00         0.00 N vcl total
           4         0.00         0.00 N vcl available
           1          .            .   N total active purges
           1         0.00         0.00 N new purges added

---

Next step: deploy on our last server, and really blocking !