Varnish cache – What is it and How it works?

Varnish Cache is a specialized caching solution designed to optimize website speed, serve files and handle high traffic efficiently. This article explores the fundamentals of Varnish Cache, its workings, and its benefits for modern web applications.

What Is Varnish Cache?

Varnish Cache is an open-source HTTP accelerator, also known as a reverse proxy. It is designed to speed up web applications and websites by caching (storing) copies of frequently accessed web pages and serving them directly to users without querying the backend server. Originally released in 2006, Varnish has become a trusted solution for handling high traffic volumes and improving website performance.

Key Features of Varnish

• High Performance: Capable of handling millions of requests per second.
• Flexibility: Configurable using the Varnish Configuration Language (VCL).
• Backend Agnostic: Works with various web servers like Apache, Nginx, and others.
• Advanced Caching Mechanisms: Supports fine-grained control over caching policies.

How Does Varnish Cache Work?

At its core, Varnish Cache acts as an intermediary between users and your web server. Here’s a step-by-step breakdown:

1. Initial Request:
   • When a user makes a request to your website, Varnish intercepts it before it reaches your web server.
2. Cache Check:
   • Varnish checks its cache to see if a copy of the requested content exists.
   • If the content is cached (“a cache hit”), Varnish serves it directly to the user, bypassing the backend server.
   • If the content isn’t cached (“a cache miss”), Varnish forwards the request to the backend server.
3. Backend Response:
   • The backend server processes the request and sends the response back to Varnish.
   • Varnish stores this response in its cache for future requests.
4. Subsequent Requests:
   • For subsequent requests for the same content, Varnish serves the cached copy, reducing server load and improving response times.

Why Use Varnish Cache?

• Speed: Dramatically reduces page load times, enhancing user experience.
• Scalability: Handles high traffic effortlessly, making it ideal for e-commerce sites, news portals, and video streaming platforms.
• Reduced Server Load: Decreases the workload on your backend servers by serving cached content.
• Customization: Allows developers to tailor caching rules using VCL.

Setting Up Varnish Cache

Here’s a quick guide to setting up Varnish Cache on a Linux server:

1. Install Varnish:sudo apt update sudo apt install varnish
2. Configure Varnish:
   • Edit the VCL file, typically located at /etc/varnish/default.vcl.
   • Define your backend server:backend default { .host = "127.0.0.1"; .port = "8080"; }
3. Update Web Server Port:
   • Change your web server (e.g., Apache or Nginx) to listen on port 8080 instead of 80.
4. Start Varnish:sudo systemctl start varnish
5. Verify Setup:
   • Ensure Varnish is running on port 80 by checking its status or using a tool like curl.

Fine-Tuning with VCL

Varnish Configuration Language (VCL) allows you to define custom caching policies. For instance, you can specify which URLs to cache, set cache expiration times, or bypass caching for specific requests.

Example:

vcl 4.0;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }
    if (req.url ~ "^/api/") {
        return (pass);
    }
    return (hash);
}

sub vcl_backend_response {
    if (beresp.ttl < 1s) {
        set beresp.ttl = 10s;
    }
}

sub vcl_deliver {
    set resp.http.X-Cache = "HIT";
}

Explanation of Example

• Backend Definition: The backend server is configured to listen on 127.0.0.1:8080.
• Request Handling: Requests that are not GET or HEAD or are directed to /api/ bypass the cache and are sent to the backend.
• Backend Response Handling: Ensures all responses have a minimum Time-to-Live (TTL) of 10 seconds.
• Custom Headers: Adds a X-Cache header to responses to indicate they were served from the cache.

Monitoring Varnish Cache

To effectively monitor and manage Varnish Cache, several tools and techniques can provide detailed insights into its performance and help troubleshoot issues. Here’s how you can expand on the topic:

1. Using varnishstat:
   • varnishstat provides real-time statistics about the performance and health of your Varnish cache.
   • Key metrics include:
     • Cache hits and misses: These indicate how efficiently Varnish is serving cached content.
     • Request rates: Track client and backend request rates to identify traffic patterns.
     • Backend health: Monitor backend availability and performance metrics.
   • Regularly review these stats to detect anomalies or trends that may require attention.
2. Using varnishlog:
   • varnishlog logs all HTTP transactions passing through Varnish, offering detailed insights into request handling.
   • Use it to:
     • Analyze specific requests for debugging purposes.
     • Identify patterns in cache misses or backend fetches.
     • Debug configuration issues with Varnish Configuration Language (VCL).
3. Leveraging varnishtop:
   • This tool displays the most frequent transactions in Varnish.
   • Identify hot URLs, frequently accessed headers, or common request methods to optimize caching strategies.
4. Using varnishncsa:
   • Generates logs in the NCSA (Common Log Format), which can be integrated with log analysis tools like ELK Stack for broader analytics.
   • Helpful for integrating Varnish monitoring with existing log management systems.