Cache invalidation to automatically update all cached template content?

18

2

I am using the cache tag in my templates. How can I assure that a site visitor always gets the data directly from the cache and never has to wait for it to be generated?

My cache gets invalidated quiet often, as this happens each time new content is created or edits are made.

Any idea on how to trigger the recreation of all cached template content automatically?

.

Edit:

To better explain my use case: some of my craft.assets and craft.entries calls result in a lot of DB queries. Without the template being cached this leads to 3+ seconds delay in page-load. Using Craft cache this is solved for all but one site visitors, because there is still the first visitor, that has to trigger the caches to be created.

I think you shouldn't ignore this single user and hope someone comes up with a solution as additional 3+ seconds in page-load can easily cause customer loss. Further I'd appreciate a solution feasible with core functions, as I consider performance being a key component to every site.

.

2nd Edit:

The reason I started a bounty on this question was to get more attention on it from the community and also from P&T. This seems to work out so far, so thank you all for your thoughts on this and for having this interesting discussion, which is sometimes a little too technical for me (who's not a php/server developer). I actually don't know what to do which the bounty points, but for me it was already worth spending them (who cares for pts anyway?)!

carlcs

Posted 2014-06-19T14:10:06.760

Reputation: 31 320

2

I think it'd be ideal if a cached file never expired and the 'cacheDuration' config option was really just an interval in which the cached file was regenerated using a background action using something like http://www.yiiframework.com/extension/runactions/. This way, every user is served a cached file of some sort except user who hits the page for the very first time.

– Wes Rice – 2014-06-20T00:01:09.883

Sounds good to me, @Wes! Will make this a feature request if there is no answer. – carlcs – 2014-06-20T07:18:08.753

1Why not a plugin that empties the cache and crawls the front end to re-trigger caching, tied to cron? – Matt Stein – 2014-06-24T16:59:48.420

I'm confused about the intention here. You want to use the {% cache %} tag and you don't ever want it to be invalidated? How will users see if a new entry has been posted or updated? Or maybe I'm missing something. Maybe update the original question with some more context for your specific use case. – Brad Bell – 2014-06-24T22:07:08.533

2@Brad I don't want it to be not invalidated, but I want the caches to be updated directly on invalidation without a site visitor to trigger it and thereby has to wait for the creation. – carlcs – 2014-06-24T22:12:05.060

1This is a tricky topic, but one that I have found to be a frequent request with heavily cached sites. CE Cache (ExpressionEngine), has an option "refresh" cache upon clearing, asynchronously.

Ideally how I think this should work is: – Tim Kelty – 2014-06-25T13:18:12.567

1Ideally, how I think this should work is:

  • Cache is determined to be expired
  • Before the cache is deleted, a request is made, somehow qualified to get the un-cached copy, while the normal, public-facing request stays cached.
  • Once the new data is received, replace the existing contents.

In theory, this should allow for cache expiration and refreshing without any un-cached downtime. However, I've yet to see a system that works like this. – Tim Kelty – 2014-06-25T13:24:28.860

Thanks @Tim for your input. Started to think I am the only one seeing a reason to do sth. like this. – carlcs – 2014-06-25T13:36:28.597

on the points, you could give them to Bryan Redeagle. In spite of the way popularity works, it's his response that generated the best comments, and he should have been voted up anyway though he may be more careful about response to negative answers in the future ;) – narration_sd – 2014-06-25T23:30:06.100

Answers

4

I built the Cache Warmer plugin recently, which takes a blanket approach to refreshing your pages cache, by getting entries url with the .getUrl() method, and then send of batch requests to those url's.

I'm currently using it together with the Cache Clear plugin, when deploying changes to our site, and it's working well. This is a very simplistic approach to a potentially complex problem, so I would say a more granular cache invalidation/refresh strategy might be needed, depending on your use case.

However, I think there is something to be said for this approach on small to medium sites.

I want to at least add some hooks, so that you can warm up single entries/elements as well. My plugin doesn't know which pages your cached elements are living on, but there is a big chance those pages will be hit and warmed up - as long as they're located in a section, and that section is toggled in the plugin settings.

Fred Carlsen

Posted 2014-06-19T14:10:06.760

Reputation: 2 424

Yeah, currently a plugin provided solution like Cache Warmer or Cache Monster is best.

– carlcs – 2016-02-19T18:57:21.053

2

I do not believe that this is a feasible feature request. Cache Invalidation isn't checked until Twig checks for it when it encounters the cache tag. If the content being cached is rarely updated, you can instead set the cache expiration date to something far off. Maybe something like {% cache for 3 years %}.

Bryan Redeagle

Posted 2014-06-19T14:10:06.760

Reputation: 3 867

"this happens each time new content is created or edits are made" != "because I set my cache duration to 1 day" – carlcs – 2014-06-22T22:38:30.277

This is true. If you look at TemplateCacheService.php it shows the functions that wipes caches base on Element Type and Element ID. It also runs it on each request. To rebuild caches, you would need to create a task that runs a pretend page request for each cached path. But that wouldn't change how the rest of the cache system works. It would still refresh based on his config, because their cache system is base on Yii's cache system. I can point out the files if you want to look at how it's all put together. I welcome a correction from P&T because they would clearly know better than I. – Bryan Redeagle – 2014-06-22T23:10:21.177

I definitely don't want P+T to change how the complete cache system works. Just looking for an answer on how to get rid of any delays site visitors may encounter because of invalidated caches. – carlcs – 2014-06-24T10:24:00.633

That task to run fake page requests you are talking, is this something that could be done with http://www.yiiframework.com/extension/runactions/ @WesRice mentioned? Wouldn't something like this, run automatically after each saveEntry solve my problem? Admittedly I don't know if craft would know what caches need to be recreated. Maybe this better has to be set manually (ie. in the sections settings).

– carlcs – 2014-06-24T10:45:01.393

If you can imagine this being a plugin, why do you think this shouldn't go into Craft core? – carlcs – 2014-06-24T10:53:49.480

I don't see this working in the core because P&T can't predict how many caches reference a particular element. It could be a few or a few hundred. There have been a few sites toting thousands of entries. If it were to go into the Core, it would have to be optional. – Bryan Redeagle – 2014-06-24T13:33:02.453

As a plugin, you could read through the entry controller file and see if the onSaveEntry event is called before or after the caches are wiped. You would still need set your cache expiry to a far off date. – Bryan Redeagle – 2014-06-24T13:35:12.390

3"If you can imagine this being a plugin, why do you think this shouldn't go into Craft core?" - I can imagine a lot of things in a plugin and 99.999% of them shouldn't be in core. – Brad Bell – 2014-06-24T21:45:13.537

@Brad Yeah, I am really looking forward to the app store! I probably worded this wrong, as I only wanted to know the reasons why this shouldn't be considered useful core functionality. – carlcs – 2014-06-24T22:55:39.333

1Not sure why you put a 200 pt. bounty on it unless hoping to shake something out of a tree. I would think such a plugin would need to be intelligent, possibly _holdingoff changes to avoid load during busy hours, and updates all in the middle of the night if you have one for span of site visitors. You see this reverses what you ask for, but may be a better strategy. I would hope for no naive plugins on a thing like this. Maybe another approach would be to put up an auto-takedown alert 'Site refreshing, will show in a moment' if you can afford busy-time update delays at all? – narration_sd – 2014-06-25T02:03:14.903

@narration I didn't imagine this to be such a complicate topic when I first asked the question, so thanks for your thoughts on it (that's why I started that bounty – to get answers like yours! Btw. what's with that tree?). But may I ask why you would take the site down while a cache recreation is executed? You expect it to have greater impact on site performance or because of conflicts with site requests? Couldn't sth. like this be a low priority task running in the background, gemütlich replacing old cache files in the db with new ones. – carlcs – 2014-06-25T14:11:26.373

Hallo Christian, 'shake sth. out of tree' is an idiomatic expression, means trying to get sth. to appear ;). Actually my idea was like yours, not taking the site down but using un-busy intervals to catch up on cache update. You're quite correct I think that this doesn't have to wait, but can be a continuous low-priority activity to get best results. That way could be most rapid and most smooth; just most complicated. Bestimmt gemütlich ;) – narration_sd – 2014-06-25T19:27:58.293

For Brad: 'acerbic' ;) And very true...! – narration_sd – 2014-06-25T19:33:33.427

1

I have a plugin that clears the cache from outside the control panel, I'm sure its possible to rebuild the cache using the same type of approach. Here is a link to my plugin if you need a starting point.

https://github.com/themccallister/craft.cache-clear

Without looking at the code, it may be possible to clear the cache and then immediate build it.

However, I agree with the majority that this should NOT be a core feature of Craft. Craft is focused on content and extensibility so adding more and more to the core is not in their, or our, best interest.

Jason McCallister

Posted 2014-06-19T14:10:06.760

Reputation: 1 604

Thanks for your ideas! And I totally agree, the focus on a select group of features is one of the reasons I love Craft so much. We definitely don't need any bloat! Bloat's what plugins are for. ;) But caching is one of those features already built into core. P&T have chosen to provide each Craft install caching functionality out of the box. A decision I really appreciate, because it makes it easy for inexperienced users (like I am one) to get a performant site up and running. – carlcs – 2014-06-25T20:43:56.387

Now that caching belongs to the core, I think that it is in the interest of P&T to make this work great (and flexible!). It's good as it is today, but it can certainly be improved in one way or another, and I actually have little doubts that caching will not be further developed down the road (Craft's is just 1 year old!)! – carlcs – 2014-06-25T20:45:08.787

0

I just came across this feature request on http://feedback.craftcms.com/ and I think this is pretty much the feature I had in mind when asking the question:

Add lazy cache rebuilding to the cache tag

PS: please don't upvote this answer, but do so on the feature request. :D

carlcs

Posted 2014-06-19T14:10:06.760

Reputation: 31 320