Skip to content

File System Monitor (fsmon)

The File System Monitor (fsmon) code repository contains files that will build a Docker image that can be used to monitor a directory and emit events when file systems events happen. It will either use inotify or fswatch to do the monitoring depending on how it is configured when it is run. You will specify the directory on the host machine to monitor by mounting the directory as a volume in the container. The Docker image for this service is located here:

MBARI fsmon Docker Image

It POSTs file event messages to an HTTP endpoint and while you are welcome to write your own service to handle the events, a handler was written to deal with these events and it hosted at the BitBucket repo here:

MBARI fsmon-handler Repo

and the corresponding Docker image is located here:

MBARI fsmon-handler Docker Image

Data Flow

graph LR repo-1[repo-1 directory] --> fsmon-1 repo-2[repo-2 directory] --> fsmon-2 fsmon-1 -- json --> uri-1[http://host1/endpoint
see fsmon-handler] fsmon-1 -- json --> uri-2[http://host2/endpoint
see fsmon-handler] fsmon-2 -- json --> uri-1[http://host1/endpoint
see fsmon-handler] fsmon-2 -- json --> uri-3[http://host3/endpoint
see fsmon-handler] click uri-1 href "https://docs.mbari.org/fsmon-handler" "fsmon-handler documentation" click uri-2 href "https://docs.mbari.org/fsmon-handler" "fsmon-handler documentation" click uri-3 href "https://docs.mbari.org/fsmon-handler" "fsmon-handler documentation"

Building

In order to build the docker image for this service, you would run

docker build .

You can optionally tag the build so that you can choose the name of the image for easier use later when you want to run it. Use the -t flag for this:

docker build -t mbari/fsmon:x.x.x .

Note that this build take a REALLY long time and it's easier to use the image from DockerHub.

Running

If you build the image as above or use the image from DockerHub, you can then run the fsmon service and configure it using the environment variables below.

Environment Variables

Note that you need to create a file named .env with the environment variables necessary (or specify them on the command line). You can copy the env.template file and uncomment and edit the variables that you want to specify. The environment variables are:

  • FSMON_BASE_URL: (required) This is an HTTP(S) URL that points to the base directory being monitored. This doesn't necessarily have to be valid, but if you want the resource URL for the files that are being monitored to be correct, this must point to the HTTP location of the base directory that is being monitored by this application. The file paths will be appended to this base URL.
  • FSMON_ENDPOINT_URLS: (required) This is a comma separated list of URLs where the events generated by this service will be sent via HTTP POST.
  • FSMON_REPO_NAME: (required) This is an arbitrary name that you specify to give some kind of name space, or context, to the messages being sent to the FSMON_ENDPOINT_URLS. The service receiving these messages won't have any kind of context for these messages, so it's important to give that to them using this variable. If you leave it blank, a combination of the HOSTNAME and the directory name will be used for one, but this is usually NOT what you want. Maybe it's a project or repository name. It should be all lowercase, no special characters, and no spaces (use dashes in place of spaces).
  • FSMON_MONITOR_METHOD: (optional) This is either 'inotify' or 'fswatch' and defaults to 'inotify' if you do not specify.
  • FSMON_FSWATCH_MONITOR: (optional) If you choose to use fswatch as the monitor method, you can use this variable to override whatever default fswatch will choose to use to monitor a directory. You can consult the documentation for fswatch to find out more, but this is normally used so that you can choose the 'poll_monitor' if the directory being monitored is not throwing events at the kernel level (like an NFS mount).
  • FSMON_FILTER_PATTERNS: (optional) This is a comma separated list of regular expressions that will be used as matches against the full path of the file to see if it should be skipped. If the match is successful, no events will be published related to this file or directory.
  • FSMON_LOG_LEVEL: (optional) This is either 'INFO' or 'DEBUG' and sets the verboseness of the logging. INFO is the default.
  • FSMON_DIRECTORY_TO_MONITOR: (optional) This is the directory that is to be monitored and remember, this is RELATIVE TO THE CONTAINER. It defaults to /data/repo and all you need to do is mount your host directory to monitor to that location and it will work. If for some reason you want to change the location to monitor inside the container, this is the variable to use.

Volumes

The volume you want to mount is the directory that you want monitored on your host and it should be mounted at the /data/repo location in the container (unless you've changed the FSMON_DIRECTORY_TO_MONITOR environment variable - see above). This can be done using the -v flag at the command line:

-v /full/path/to/host/directory/to/be/monitored:/data/repo

Example

An example of running this service could be:

docker run --env-file .env -v /host/dir/to/monitor:/data/repo mbari/fsmon:x.x.x

Testing

When the container starts up, it starts a small server internally that will simply log the messages being sent to to the console. If you add the URL:

http://localhost:3000

to your list of endpoint URLs, you will see the requests and message bodies show up in the console.

Development

If you are working on the code in this repository, there are some steps you will need to take once your work is complete, tested, etc. When you are ready to deploy the changes you need to do the following:

  1. Come up with a version number that makes sense for the scope of changes you have made to this codebase. If they are bug fixes, you typically increment the third number in the version, if they are feature enhancements or additions that are not breaking changes for clients of this code, you can increment the second number in the version. If the code changes are substantial and/or will break client usage, you should increment the first digit in the version number.
  2. Add a line at the bottom of this file describing the changes that are associated with this new version number.
  3. Commit your changes into your local Git repository.
  4. Create a Git tag in your local repository with the number you generated in the first step.
  5. Push the HEAD to the remote repository
  6. Push the tag you created to the remote repository
  7. Next, you should generate a new Docker image that can be pushed to the remote DockerHub repository. You can build this by running the following (where x.x.x is the version number you generated in step one.):

    ```bash
    docker build -t mbari/fsmon:x.x.x .
    ```
    
  8. Once the image is done, push it to DockerHub using:

    ```bash
    docker push mbari/fsmon:x.x.x
    ```
    

where x.x.x is the version number you generated in step one.

Example message structure

Create File

{
    "timestamp" : "1549322745",
    "event" : "Created",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/file1.txt",
    "resourceType" : "FILE"
}

Create Directory

{
    "timestamp" : "1549322848",
    "event" : "Created",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/dir1",
    "resourceType" : "DIRECTORY"
}

Add File

{
    "timestamp" : "1549322892",
    "event" : "Updated",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/dir1/file2.txt",
    "resourceType" : "FILE"
}

Add Directory

{
    "timestamp" : "1549322927",
    "event" : "Created",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/dir1/dir2",
    "resourceType" : "DIRECTORY"
}

Remove/Delete File

{
    "timestamp" : "1549322986",
    "event" : "Removed",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/dir1/file2.txt",
    "resourceType" : "UNKNOWN"
}

Remove/Delete Directory

{
    "timestamp" : "1549323012",
    "event" : "Removed",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/dir1/dir2",
    "resourceType" : "UNKNOWN"
}

Change/Edit File

{
    "timestamp" : "1549323052",
    "event" : "Created",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/file1.txt",
    "resourceType" : "FILE"
}
{
    "timestamp" : "1549323052",
    "event" : "Updated",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/file1.txt",
    "resourceType" : "FILE"
}

Move File

{
    "timestamp" : "1549323126",
    "event" : "Removed",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/file1.txt",
    "resourceType" : "UNKNOWN"
}
{
    "timestamp" : "1549323127",
    "event" : "Updated",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/dir1/file1.txt",
    "resourceType" : "FILE"
}

Move Directory

{
    "timestamp" : "1549323183",
    "event" : "Removed",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/dir2",
    "resourceType" : "UNKNOWN"
}
{
    "timestamp" : "1549323183",
    "event" : "Created",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/dir1/dir2",
    "resourceType" : "DIRECTORY"
}

Rename File

{
    "timestamp" : "1549323220",
    "event" : "Removed",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/dir1/file1.txt",
    "resourceType" : "UNKNOWN"
}
{
    "timestamp" : "1549323221",
    "event" : "Updated",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/dir1/file2.txt",
    "resourceType" : "FILE"
}

Rename Directory

{
    "timestamp" : "1549323260",
    "event" : "Removed",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/dir1/dir2",
    "resourceType" : "UNKNOWN"
}
{
    "timestamp" : "1549323260",
    "event" : "Created",
    "repoName" : "localhost-test",
    "resourceURL" : "https://myhost/data/repo/dir1/dir3",
    "resourceType" : "DIRECTORY"
}

Notes

  • The reason there are two methods is because inotify does not work with NFS mounted drives and fswatch can be set up to poll to fix that issue. However, on my Mac, a network drive seems to behave the same, so I need to do this on a linux box that has an NFS mount to test this behavior.
  • When a file is 'moved into' the directory structure that is being monitored, the message is that it was 'Updated', not 'Created', but it shows as 'Created' if it was a directory.
  • When a file is changed, it is read as two separate events, a 'Created' followed by an 'Updated', but with the same timestamp.
  • When a file or directory is moved (but still within monitored directory structure), two events are issued: a 'Removed' from the old location and a 'Created' at the new location.

Version notes

  • v0.0.2 - removed the check for repository names longer than 15 characters as I don't think that is a restriction anymore.
  • v0.0.3 - just an update to documentation really.