fs-sentinel

fs-sentinel is a (currently) Linux service that actively observes file systems for modifications, allowing clients to easily check if a given file system has changed, and reset the “changed” status when they choose.

Motivation

When maintaining incremental file system snapshots on a system with pruning to limit the number of retained snapshots, duplicate snapshots should be avoided at all costs. Take the following scenario:

• In our pruning configuration, we specify that we want to keep 5 hourly snapshots for a given file system (note that this is a platform/implementation-specific term)
• We take an hourly snapshot and reach our hourly snapshot capacity; if we were to take a new snapshot, it would replace our oldest hourly snapshot.
• However, for the next 5 hours, our file system does not change at all
• If our automated snapshotting continued over this period, we would end up with 5 equivalent snapshots, which are useless compared to the 5 unique snapshots from the hours prior

To prevent this scenario, we need a way to efficiently determine if a snapshot should be taken for any given file system, which our snapshotting service could hook into to guard against duplicate snapshots. This is where fs-sentinel comes in!

Introduction

fs-sentinel is a service which actively monitors file systems* for changes, keeping track of which ones were modified since the last mark operation and allowing other programs to easily check if a file system has changed. File system* in this context is an abstract term, determined by the underlying Platform implementation. The stock Linux implementation uses the Linux definition for “file systems”, which typically consists of individual disk partitions and volume manager subvolumes. This implementation uses the fanotify API, specifically the fsnotifywait command-line program, to efficiently monitor file systems for changes.

The project consists of a library and CLI component. The library exposes the daemon implementation and a Rust trait which abstracts all platform-specific behavior, allowing you to use the daemon logic for non-Linux platforms, as well as applying it to “file system” definitions which differ from the stock implementation. For example, an implementor could trivially treat directories within one physical file system as separate file systems, so long as they specify how to monitor those directories.

CLI Usage

Note: see INSTALL.org for installation instructions.

USAGE:
    fs-sentinel <SUBCOMMAND>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

SUBCOMMANDS:
    check            If the specified filesystem has been modified, returns 0, otherwise returns 1
    daemon           Runs the daemon with a specified list of monitored filesystems
    help             Prints this message or the help of the given subcommand(s)
    list-modified    Yields a newline-delimited list of filesystem ids corresponding to all modified filesystems.
    mark             Resets the specified filesystem to unmodified

Here’s an example on how you might use the daemon. For the sake of reproducibility, let’s create an artificial file system configuration of tmpfs’s to use in the example.

$ mkdir /tmp/exampleRoot && cd /tmp/exampleRoot
$ for dir in foo bar baz; do mkdir $dir; sudo mount -t tmpfs "fs-sentinel_$dir" $dir; done

First, let’s get a list of all the unique “file systems” that we have on our system. Each of these TARGET’s are valid paths to pass to the CLI.

$ findmnt --output target,source,fstype # heavily truncated output
TARGET                         SOURCE             FSTYPE
/                              /dev/nvme0n1p1     ext4
├─/tmp                         tmpfs              tmpfs
│ ├─/tmp/exampleRoot/foo       fs-sentinel_foo    tmpfs
│ ├─/tmp/exampleRoot/bar       fs-sentinel_bar    tmpfs
│ └─/tmp/exampleRoot/baz       fs-sentinel_baz    tmpfs
├─/pool                        pool               zfs
│ ├─/pool/var                  pool/var           zfs
│ ├─/pool/arch-home            pool/arch-home     zfs
│ └─/pool/general-store        pool/general-store zfs
└─/nix                         pool/var[/nix]     zfs

Notice that though the directories corresponding to the different TARGET’s are nested, the actual filesystems (identified by the SOURCE’s) are completely distinct! This means that creating a file /tmp/x would result in the filesystem corresponding to /tmp being marked as Modified but would not affect the root / filesystem.

Now, let’s set our sentinel to watch the file systems pertaining to the following directories: /tmp/exampleRoot/foo, /tmp/exampleRoot/bar and /tmp/exampleRoot/baz. We will arbitrarily give each of these filesystems the identifiers: foo123, bar456 and baz789, just to illustrate that these identifiers can be different from both the TARGET’s and the filesystem SOURCE’s (from the findmnt output).

$ sudo fs-sentinel daemon foo123=/tmp/exampleRoot/foo bar456=/tmp/exampleRoot/bar baz789=/tmp/exampleRoot/baz

Note that sudo is not always required to be able to watch a file system; notably, if the root of the mount is owned by your own user, fs-sentinel can generally monitor it even without root permissions. However, there are certain edge cases that come up when using SELinux and/or attempting to use fs-sentinel from inside of a Docker container, where permissions are more complex than simple file mode bits. These complex use cases are unfortunately unsupported for the time being.

Now that we’ve launched the daemon, in another terminal, we can see it in action. Let’s start off by getting the list of currently “modified” file systems!

$ sudo fs-sentinel list-modified

The output is empty, since we haven’t touched any of the directories yet!

Now, let’s modify /tmp/exampleRoot/foo and see that fs-sentinel immediately picks it up!

$ sudo touch /tmp/exampleRoot/foo/randomfile
$ sudo fs-sentinel list-modified
foo123

If we check the status of foo123, the command will return exit code 0, which means that the file system has been modified.

$ sudo fs-sentinel check foo123 && echo "foo123 was modified!"
foo123 was modified!

On the other hand, check‘ing the status of either of the other filesystems will return exit code 1, meaning that the file system has not been modified.

$ sudo fs-sentinel check bar456 && echo "(This will not print)"
$ sudo fs-sentinel check baz789 || echo "It was /not/ modified!"
It was /not/ modified!

Next, let’s mark our foo123 file system to reset its status to UnModified.

$ sudo fs-sentinel mark foo123

If we re-~check~ the status of foo123, we would see that it now reports exit code 1 instead. Also, as you might expect, list-modified is now empty again.

$ sudo fs-sentinel list-modified

You can stop the daemon by sending Ctrl-C to the attached tty, or by sending a SIGTERM to the process. Note that when stopping the daemon gracefully, it will cache its current list of Modified file systems, so that after you relaunch the daemon, these same file systems will retain their Modified status!

As an aside, it’s recommended that rather than running the daemon manually, you write a Systemd unit to manage it. An example unit file is provided in /zfs-linux, which will be elaborated on in a later section.

This should be sufficient to adapt fs-sentinel to your own use cases; however, if something in the documentation is unclear, please file an issue and I’ll do my best to clear it up!

Provided Setups

Here are some examples on how you might use fs-sentinel.

ZFS+Sanoid Linux Integration

fs-sentinel comes with short example code for monitoring mounted ZFS datasets on system startup, as well as example implementations for hooks to be used with the amazing sanoid project: my primary use case going into development.

To use the project with sanoid simply point the pre_snapshot_script and post_snapshot_script in your Sanoid configuration to the corresponding scripts from ./zfs-linux. To use this setup, consider running make sanoid-install, which will place the relevant scripts and binaries in /usr/local/bin by default, as well as install a Systemd service for launching the daemon. This service will watch every single mounted dataset After that, modify your sanoid.conf and you should be good to go!