How do I configure Varnish to work with Craft?

28

7

Has anyone successfully used Varnish with a Craft build? I'm trying to set up Varnish on my server but I'm very new at this. Can anyone share their config file or point me in the right direction? We have this build on an EC2 instance and so far it works great but we'd like to optimize it further.

Carlo Laitano

Posted 2014-06-12T00:14:44.180

Reputation: 241

Answers

15

Ok, Carlo, I'll get in this improved answer tonight, anyway, probably cut down and informate the other answer in morning here, so comments stay. I've got a sick animal here house-sitting for brother & family, why I'm a little slow.

To accelerate what you have, it really depends on the content - whether you have logins or other personalization among your high ten-thousands of daily hits. If you don't, it may be possible to get some quick improvement, and then move up through the other things. Throughout this I'm going to be thinking of a big sports site I've seen you have, thinking it may have timely content updates while not personalization; but I don't remember it with assurance on that detail.

  • Digging into CloudFlare, since you do have it there on EC2, I find I was mistaken about what it does before you may configure it. This also explains some things I thought I saw in earlier brief shakedown tests myself.

  • With automatic, stock configuration, CloudFlare only edge caches media referred to, like jpgs, js, css etc. -- not html of any kind, to keep things safe for first users. There are three further levels of caching possible, which includes the ability to do whole pages. You use PageRules to enable this, and can for a whole site depending on the match filter. If you don't have personalization, I think you could get a huge benefit right away by setting this up, rather than using Railgun. Or for some pages, anyway? So this is what I'd look into first, if the content fits. CloudFlare doc link here. You can set time to live, so as to have essential static pages which still update regularly if you have timely but not personalized content.

  • If you have personalization of pages, then you do indeed need server-side caching to rein in such large page demands, I would agree. Railgun working as intended is only going to help with cutting out handshake latency, compressing, and so forth. This is great, but it still requires that you run the (Craft) CMS each time someone requests a page, in order to see if there is a delta Railgun would send.

The next stage then I would think would be to enable Craft's own caching, to cut Craft's overhead way down, as this is very effective, targetable to what should and shouldn't be cached in your site, and can use EC2's ElastiCache, w/doc link here, to get you memory-resident caching's speed. Craft's /app/etc/config/defaults has scripts for both Redis and Memcache(d) setup, which I think you just copy into your /craft/config folder once the facility is up on your EC2 machine.

With fortune, this could be an easy solution to deal with, again if you need to have personalized or otherwise dynamic content, and you can try it out for gain before any ElastiCache by just turning it on in the file or db modes, for which there are also default scripts in the same location.

I'd read very carefully the information that's in Victor In's link reference, as it gives quite a bit of information straight from Brandon (and with others' interpreting) compared to present doc. I agree with what he says about effectiveness of using db caching from experience before, as databases tend to have very efficient caching for their own operation, including sorting out high numbers of requests. What he says about the db as basis for the great effectiveness of Craft's detailed cache breaking with content updates is also very astute, and one of those things which makes Craft very intelligent with how it handles your content.

What about Varnish, then? Well, it does look very powerful and proven; also requiring the most detail of any speedup to configure. Setting it up in the first place is simple; you just set your webserver to run on a non-visible port, and tell Varnish where to find it while arranging Varnish to handle your visible standard http and possibly https ports. That installs the engine, as in their pretty good-looking Varnish doc with a kind of guide on this link.

I don't remember if EC2 comes with a C compiler installed, but easy to do, and this will be necessary to compile the VCL scripts you configure Varnish with to know your page requirements. VCL itself? Again, there's good doc, but you are really going to want to know exactly what your pages need, if they need personalized results. This is really about the site design, not anything about Craft itself, and here we are considering the original question directly.

In summary, I think that depending on your content, you have three levels you can go through here to reduce load and increase apparent speed for your site.

  1. If you are so fortunate as to have non-login and otherwise non-personalized content, you can probably do the large arrangement by configuring CloudFlare to actually edge-cache the html pages as well as media content. This will give you essentially static site page speed, also hosted out near many visitors (if you're lucky on CloudFlare sites), plus having off-loaded your page request handling out to those edge sites. Using embedded monitoring may keep you lucky on getting enough access data to satisfy the client.
  2. If your client needs logins or personalization of other kinds, then one of the levels of Craft caching (and selection of what to cache) is going to help a lot. This would really take advantage of Railgun, while feeding it the information it needs very efficiently for your EC2 instance(s). This might be enough, and might see Railgun operating as you'd expect it to reliability-wise. Again, you could start with a file- or db- level Craft cache configuration, and then after working up redis or memcached on a staging server probably, fitting that in for even higher efficiency and speed.
  3. Varnish itself, then. I seem to feel this would be the most effort to configure, but if needed, give the highest effectiveness, if you need to go that far. The configuration itself is about your static vs. dynamic page content design, not Craft itself, so you can use such sources as stackoverflow's varnish tag to get hints or questions answered on it. Here's a link to an interesting start for that, mentioning relationships between varnish and memcached in the frame as I've put some thoughts here. ElastiCache-driven native Craft caching will give you the highest performance for necessary dynamic content here, and it looks like Railgun will improve on Varnish alone, by getting that fast-resolved dynamic content out to the edges in much less time than a standard internet request would do so.

Carlo, hope it's been helpful to have another head looking at this, and that possibly these stages in some form may give you ideas to do a useful rapid response, then refine. Craft community advancing on many fronts at the moment, anyway ;)

narration_sd

Posted 2014-06-12T00:14:44.180

Reputation: 1 512

This helps a lot! I'll look into it in more detail and see what we can do with all these stages. Load-balancing is tricky and caching has been very delicate, but I'm sure we're getting closer to an ideal setup. – Carlo Laitano – 2014-06-18T07:31:40.253

Sounds great - will be very interested to hear how it goes! – narration_sd – 2014-06-18T08:03:19.583

4

[Edit: I'm deprecating this answer, having a better one up now I believe, but leaving it as it's attracted very useful comments. I think what's said here isn't quite wrong, but has been considerably improved upon, and can be more so as the community gains experience.

In particular, though for good reason, Railgun has turned out to be more naive than I thought (or see it can be configured at present), so that it is a great help on responsiveness out at the edge, but limited to as far as the original server is also speedy. This speed can be obtained in degrees, using Craft's caching at appropriate levels, and eventually by Varnish if one needs the ultimate available. Any of these improvements will also proportionately reduce load (and expense) on the originating server, while naturally increasing the precision required of configuration. Getting something in place sounds like it could give the space to work out such details on a staging server.]

My thought on the original question was to wonder how often Varnish was actually going to be needed. It turns out though that each of the types of caching could answer a given situation on their own, but also contribute when several are stacked, up to where there may be the additional need of Varnish,.

We have Craft's controllable and auto-breaking caching, which can engage memory-store methods like memcached or redis if needed above the effectiveness of database caching, if needed above the effectiveness it can give with just the filesystem method, especially with modern hosting having SSD or cache-SSD disks.

Beyond that, though, are the abilities of Railgun, a new addition on CloudFlare. This gives you edge caching, even on pages that may include active changes. They do this by sending only deltas, and holding your cache on edges near your client, in parallel with the static edge cashing CloudFlare began with. There are competitors, for example Incapsula.

Railgun has been apparently effective in small early trials here, but for high-page-access sites, it's going to also need load reduction and response-speed improvers for the source server, so let me conclude, and suggest the discussion of those in the fresh article.

narration_sd

Posted 2014-06-12T00:14:44.180

Reputation: 1 512

I'm not completely sure but doesn't the template {% cache %} tag only store it's data in the database? I believe the options for using memcache/redis etc are only for the native caching that Craft implements elsewhere such as after parsing a twig template. – Josh Angell – 2014-06-15T07:00:26.010

@joshangell Yes, only in the db: http://craftcms.stackexchange.com/a/107/177

– Victor In – 2014-06-15T16:25:40.320

@narraion_sd We do have Cloudflare set up for this site/server, but Railgun was causing some real odd caching issues.

This is a really high traffic website. 85k pageviews yesterday, for example. 285k pageviews in the last week. It's been responding well since we're using load balancing with EC2, but we'd like to reduce the amount of slaves necessary to handle all the traffic. – Carlo Laitano – 2014-06-17T02:14:50.537

@CarloLaitano, I see why you're interested in Varnish. I've looked into some things about it, enabling memory cache for Craft on EC2, and important details on CloudFlare. May be able to suggest a sensible path on what you're up against, and I'll do that in another answer to get it out of this commenting system, also modifying the first one because it was a little fast and loose. That link from Victor In on Josh's view is very valuable as a reference, on the Craft side. Brandon comes through again... – narration_sd – 2014-06-18T04:13:05.167

4

As requested, here's a Varnish (3.x) configuration we use: https://gist.github.com/timkelty/35625a90a6819c8f7927

This is one setup for ExpressionEngine, but the only thing there really specific to that is the EE_PURGE request portion, which makes it work with: https://github.com/kevincupp/purge.ee2_addon

We have it set up so site.com:8080 is the un-varnished port, so you can check updated content with necessarily clearing the Varnish cache.

Tim Kelty

Posted 2014-06-12T00:14:44.180

Reputation: 1 871

Do you have a 4.x config for varnish? – mbalparda – 2014-09-24T11:53:58.573