Erratic Behaviour from .NET MemoryCache Expiration Demystified

Standard

On a recent project I experienced first-hand, how the .NET MemoryCache class, when used with either absolute or sliding expiration, can produce some unpredictable and undocumented results.

Sometimes cache items expire exactly when expected… yay. But mostly, they expire an arbitrary period of time late.

For example, a cache item with an absolute expiry of 5 seconds might expire after 5 seconds but could just as likely take up to a further 20 seconds to expire.

This might only be significant where precision, down to a few seconds, is required (such as where I have used it to buffer / throttle FileSystemWartcher events) but I thought it would be worthwhile decompiling System.Runtime.Caching.dll and then clearly documenting the behaviour we can expect.

When does a cache item actually expire?

There are 2 ways your expired item can leave the cache:

  • Every 20 seconds, on a Timer, it will pass through all items and flush out anything past its expiry
  • Whenever an item is accessed, its expiry is checked and that item will be removed if expired

This goes for both absolute and sliding expiration. The timer is enabled as soon as anything is added to the cache, whether or not it has an expiration set.

Note that this is all about observable behaviour, as witnessed by the bemused debugger, because once an item has past its expiry, you can no longer access it anyway – see point 2 above, where accessing an item forces a flush.

Just as weird with Sliding Expiration…

Sliding expiration is where an expiration time is set, the same as absolute, but if it is accessed the timer is reset back to the configured expiration length again.

  • If the new expiry is not at least 1 second longer than the current (remaining) expiry, it will not be updated/reset on access

Essentially, this means that while you can add to cache with a sliding expiration of <= 1 second, there is no chance of any accessing causing the expiration to reset.

Note that if you ever feel the urge to avoid triggering a reset on sliding expiration, you could do this by boxing up values and getting/setting via the reference to the object instead.

Conclusion / What’s so bewildering?

In short, it is undocumented behaviour and a little unexpected.

Consider the 20 second timer and the 5 second absolute expiry example. When it is actually removed from the cache, will depend on where we are in the 20 seconds Timer cycle; it could be any time period, up to an additional 20 seconds, before it fires, giving a potential total of ~ 25 seconds between actually expiring from being added.

Add to this, the additional confusion you’ll come across while debugging, caused by items past their expiry time being flushed whenever they are accessed, it has even troubled the great Troy Hunt: https://twitter.com/troyhunt/status/766940358307516416. Granted he was using ASP.NET caching but the core is pretty much the same, as System.Runtime.Caching was just modified for general .NET usage.

Decompiling System.Runtime.Caching.dll

Some snippets from the .NET FCL for those wanting a peek at the inner workings themselves.

CacheExpires.cs

FlushExpiredItems is called from the TimerCallback (on the 20 seconds) and can also be triggered manually via the MemoryCache method, Trim. There must be interval of >= 1 second between flushes.

Love the goto – so retro. EDIT: Eli points out that it might just be my decompiler!

MemoryCacheEntry.cs

UpdateSlidingExp updates/resets sliding expiration. Note the limit MIN_UPDATE_DELTA of 1 sec.

MemoryCacheStore.cs

See how code accessing a cached item will trigger a check on its expiration and if expired, remove it from the cache.

FileSystemWatcher vs locked files

Standard

Git repository with example code discussed in this article.


Another Problem with FileSystemWatcher

You’ve just written your nice shiny new application to monitor a folder for new files arriving and added the code send that file off somewhere else and delete it. Perhaps you even spent some time packaging it in a nice Windows Service. It probably behaved well during debugging. You move it into a test environment and let the manual testers loose. They copy a file, your eager file watcher spots the new file as soon as it starts writing and does the funky stuff and BANG!… an IOException:

The process cannot access the file X because it is being used by another process.

The copying had not finished before the event fired. It doesn’t even have to be a large file as your super awesome watcher is just so efficient.

A Solution

  1. When a file system event occurs, store its details in Memory Cache for X amount of time
  2. Setup a callback, which will execute when the event expires from Memory Cache
  3. In the callback, check the file is available for write operations
  4. If it is, then get on with the intended file operation
  5. Else, put it back in Memory Cache for X time and repeat above steps

It would make sense to track and limit the number of retry attempts to get a lock on the file.

Code Snippets

I’ve built this on top of the code discussed in a previous post on dealing with multiple FileSystemWatcher events.

Complete code for this example solution here

When a file system event is handled, store the file details and the retry count, using a simple POCO, in MemoryCache with a timer of, something like 60 seconds:

A simple POCO:

In the constructor, I initialised my cache item policy with a callback to execute when these cached POCOs expire:

The callback itself…

  1. Increment the number retries
  2. Try and get a lock on the file
  3. If still lock put it back into the cache for another 60 seconds (repeat this MaxRetries times)
  4. Else, get on with the intended file operation

Other ideas / To do….

  • Could also store the actual event object
  • Could explore options for non-volatile persistence
  • Might find sliding expiration more appropriate in some scenario

A robust solution for FileSystemWatcher firing events multiple times

Standard

Git repository with example code discussed in this article.


The Problem

FileSystemWatcher is a great little class to take the hassle out of monitoring activity in folders and files but, through no real fault of its own, it can behave unpredictably, firing multiple events for a single action.

Note that in some scenarios, like the example used below, the first event will be the start of the file writing and the second event will be the end, which, while not documented behaviour, is at least predictable. Try it with a very large file to see for yourself.

However, FileSystemWatcher cannot make any promises to behave predictably for all OS and application behaviours. See also, MSDN documentation:

Common file system operations might raise more than one event. For example, when a file is moved from one directory to another, several OnChanged and some OnCreated and OnDeleted events might be raised. Moving a file is a complex operation that consists of multiple simple operations, therefore raising multiple events. Likewise, some applications (for example, antivirus software) might cause additional file system events that are detected by FileSystemWatcher.

Example: Recreating edit a file in Notepad firing 2 events

As stated above, we know that 2 events from this action would mark the start and end of a write, meaning we could just focus on the second, if we had full confidence this would be the consistent behaviour. For the purposes of this article, it makes for a convenient examples to recreate.

If you edited a text file in c:\temp, you would get 2 events firing.

Complete Console applications for both available on Github.

A robust solution

Good use of NotifyFilter (see my post on how to select NotifyFilters) can help but there are still plenty of scenarios, like those above, where additional events will still get through for a file system event.

I worked on a nice little idea for utilising MemoryCache as a buffer to ‘throttle’ additional events.

  1. A file event ( Changed in the example below) is triggered
  2. The event is handled by OnChanged. But instead of completing the desired action, it stores the event in MemoryCache with a 1 second expiration and a CacheItemPolicy  callback setup to execute on expiration.
  3. When it expires, the callback OnRemovedFromCache completes the behaviour intended for that file event.

Note that I use AddOrGetExisting as an simple way to block any additional events firing within the cache period being added to the cache.

If you’re looking to handle multiple, different, events from a single/unique file, such as Created with Changed , then you could key the cache on the file name and event named, concatenated.