File system crash consistency and performance

Crash consistency

POSIX does not make any useful guarantees in case of a crash (i.e., OS crash, power outage or the like), but what would we like file systems to do? A naive approach would be to perform all writes to persistent memory synchronously (i.e., wait for them to complete), and perform operations such as renames in a way that we see no inconsistent state. In case of a crash we would see the same file system state that was current (and visible to other processes) right before the crash.

While this approach may perform ok for editing, workloads that write a lot of data (e.g., untarring a file) would be slowed down a lot by this approach. So file systems usually compromise on consistency in favour of efficiency. How should they do that? There are at least two positions:

Implementation-oriented

Following this position, the application should tell the file system through the fsync() system call when it wants a specific file to be persistent. This position is taken by, e.g., Ted Ts'o, implementor of the ext4 file system. In our example, this would mean an fsync() of #F# right after writing its contents, maybe an fsync of the directory containing #F# after that, and an fsync of the directory containing F after the rename. If the application does enough fsyncs (and it's not obvious how much is enough; I am not aware of a guide for achieving consistency using fsync), the data will be consistent, if not, there may be data loss and the file system developer will blame the application and deny any responsibility.

Semantics-oriented

The file system guarantees that the persistent state represents one of the states visible in normal operation before the crash; i.e., the user may lose some updates that other processes could see before the crash, but if the application esured consistency in normal operation, the persistent state will be a consistent state. In our example F would contain either the old or the new state, no fsyncs required. However, you probably want to sync before reporting the completion of a transaction (say, confirm a purchase) to a remote user. My position is that file systems should give this guarantee. Unfortunately, last I looked among the Linux file systems, only NILFS2 gave this consistency guarantee. The advocates of the implementation-oriented approach denigrate this as O_PONIES, because they want users to think that this is unrealistic.

One advantage of the semantics-oriented approach is that an application that is debugged for consistency in case of ordinary operation is automatically consistent (albeit not necessarily up-to-date) in case of a crash.

Performance

In particular, if the application does not perform any fsyncs, the implementation-oriented file system does not guarantee any crash consitency, while the semantics-oriented file system does perform as consistently as ever.

If the application performs the fsyncs necessary for consistency on the implementation-oriented file system, the semantics-oriented file system guarantees that the persistent state represents the logical state of the file system at some point in time at or after the fsync, while the implementation-oriented file system does not give such a guarantee.

But what if we compare variants of the application tuned for the requirements of the specific file system? I.e., no fsyncs or syncs for a semantics-oriented file system if we can live with losing the progress of a certain number of seconds (e.g., when using an editor with an autosave file, getting the old file and the autosave file may be almost as good as getting an up-to-date file), but the full complement of fsyncs for an implementation-oriented file system (an application could have a flag for enabling or disabling these fsyncs).

Implementation-oriented file system

The application is going to perform a lot of fsyncs, at least as many as necessary for the consistency level that some user of the application may require, so possibly many more than needed for the user at hand. And every fsync is going to wait until the data hits persistent memory and reports back success. This may be quite slow.

Semantics-oriented file systems

For now, let's assume that the application does not need synchronicity, so it does not perform fsyncs. In that case, the file system can perform as few synchronous block device flushes (or, theoretically, asynchronous barriers) as desired to reduce the maximum time lost (e.g., one synchronous flush every 5s to guarantee that not more than 10s of work are lost): First request writes of nearly all the data and metadata to free blocks, then flush, then request a write of the commit block makes that data and metadata visible to the file system; the commit block will become persistent with the next flush at the latest, and make all the data persistent to the file system. This assumes that all writes (except maybe the commit block) go to free blocks, as happens with a copy-on-write file system, a log-structured file system, or a sufficiently journaling file system; there is some cost associated with this, but I expect that it is relatively moderate.

Even if you need to sync (e.g., before you confirm a purchase to a remote customer), this will be much rarer and cost less than the many fsyncs needed for satisfying implementation-oriented file systems. And it's clear when it is necessary to perform these syncs, because they come from application needs.

Given these issues with implementation-oriented file systems, do you really want to use one if you care for crash consistency?

In any case, keep in mind that running a given benchmark on the two kinds of file systems usually does not produce crash consistency results and therefore the performance numbers may be misleading.