Skip to content

File System Monitor Handler (fsmon-handler)

This project contains code that creates an HTTP server that is designed to accept and handle messages that come from the fsmon service. The idea is that the fsmon service watches for changes in a file system and pushes those event to something like this fsmon-handler. The fsmon-handler examines the message, and based on the way it is configured, will structure that message in JSON format and publish to AMQP and/or push the message to Slack. Since it is an HTTP server, it can be the handler for several fsmon instances.

The code in the repository is designed to be built into a Docker image that is then hosted at:

fsmon-handler docker image

To use in real environments, you would want to run this from a Docker image and configure with the environment variables listed in the next section. If you are developing code in this project, see the development section below for some notes.

Data Flow

graph LR repo-1[repo-1 directory] --> fsmon-1 repo-2[repo-2 directory] --> fsmon-2 repo-3[repo-3 directory] --> fsmon-3 fsmon-1 -- json --> uri-1[fsmon-handler
http://fsmon-handler/endpoint] fsmon-2 -- json --> uri-1[fsmon-handler
http://fsmon-handler/endpoint] fsmon-3 -- json --> uri-1[fsmon-handler
http://fsmon-handler/endpoint] uri-1 -- amqp --> exchange[Fanout Exchange] uri-1 -- json --> channel-1 uri-1 -- json --> channel-2 uri-1 -- json --> channel-3 subgraph RabbitMQ exchange --> queue-1 exchange --> queue-2 exchange --> queue-n end subgraph Slack channel-1 channel-2 channel-3 end subgraph Consumers queue-1 --> consumer-1 queue-2 --> consumer-2a queue-2 --> consumer-2b queue-n --> consumer-n end

Environment Variables

These are the environment variables that are used to configure fsmon-handler. If you are running from the Docker image, you can specify these in a file (most common way) or at the command line. If defining in a file, you can copy the env.template file as a starting point and edit it to your particular situation. The variables are:

  • LOG_LEVEL: The logging messages from this service are written to the console so that when run in a Docker environment, the can be managed with Docker's logging utilities. The LOG_LEVEL variable defines how verbose the logging utility should be. I can be one of the following:
    • error
    • warn
    • info (default)
    • verbose
    • debug
    • silly
  • SERVER_PORT: This is the port that the service will listen to and it defaults to 3000. Remember that if this is running inside a Docker container, that port is relative to the container and will need to be opened to the outside so that messages can be sent to it.
  • PUBLISH_TO_SLACK: This is 'true' or 'false' and determines if the service should try to forward the notifications to Slack. Defaults to 'false'.
  • SLACK_TOKEN: This is the client token needed to push notifications to Slack if it's been enabled. Defaults to empty but is required if Slack notifications are turned on. See note in below about how to obtain a Slack token and user id.
  • SLACK_USER_ID_TO_ADD: This is the Slack ID of the user that will be a default user to add to a channel. See note below about how to obtain a Slack user ID. This is required due to the fact that if a channel is to be created and does not exist in Slack yet, it must have a user associated with that channel before it can be created.
  • SLACK_INTERVAL_IN_SECONDS: This is an integer that defines the number of seconds between each message that is pushed to Slack. Slack has a limit on the frequency of incoming messages that I believe is one message per second. The default for this is 2 seconds.
  • PUBLISH_TO_AMQP: This is 'true' or 'false' and tells the server if it should try to foward incoming messages to an AMQP server. The default is false.
  • AMQP_HOST: This is the name of the host that AMQP messages will be sent to. It is required if you are sending messages to an AMQP server and realize it may be relative to the Docker container if running this inside Docker.
  • AMQP_PORT: This is the port that the AMQP server will be listening to. It is required if you are enabling sending messages to AMQP.
  • AMQP_VHOST: This is the VHost that will be published to if AMQP publishing is enabled. It is required.
  • AMQP_USERNAME: The username that will be used to connect to the AMQP server.
  • AMQP_PASSWORD: The password that will be used to connect to the AMQP server.

Publishing to Slack

If you would like fsmon-handler to push messages to Slack, you need to create an App in Slack and get a token. I found this page:

Slack|Getting Started

to be a really simple set of instructions on how to get a token that you can use. Also, if you think the channels that will be published to may not exist before you start this server, you will need a User ID that can be used to create new channels in Slack. To find a user's unique ID, go to your Slack application and find their name on a message in Slack and click on it. Then click on "View Full Profile", then click on the "..." (more) and then click on "Copy member ID".

Warning

One thing I found that has changed recently, is that you have to add your App's bot to the channel you want to publish messages to for this to work. Once the channel is created in Slack, you need to open Slack, go to that channel and add the bot user by using /invite @your_bot_name. There may be a way to do this programmatically at some point, it's on my list :).

Development

If you are developing code in the repository, here are some helpful hints to get you going and make development easier. There is a docker-compose.yaml file located in this top level directory that will bring up a RabbitMQ server that you can use to push events to. It will also spin up an instance of fsmon so that you can make changes in a local directory on your machine that will then cause events to be pushed to your server as you develop the code.

Before starting, find a directory on your machine that you can use as the directory that will be monitored by fsmon. For this example, I will use:

/my/host/directory/to/monitor

If you want to publish to Slack during testing, follow the instructions above to get a Slack token and user ID. To start the docker compose processes, using the directory you've identified to be monitored, start the test environment by running the following in a different shell:

export HOST_TEST_DIR=/my/host/directory/to/monitor
docker-compose up

This will start up a RabbitMQ instance as well as an instance of fsmon that is monitoring /my/host/directory/to/monitor and will push those events to your server.js process that is listening on port 3000 (which is not yet running).

Now you want to start up the fsmon-handler service and how this works will depend somewhat on how you do development, but essentially, you need to start the server by running server.js as a Node process, but you will need to configure the environment it runs in by setting the environment variables defined above somehow. This will depend on if you are running the process in a shell, in an IDE, etc. but basically the environment that server.js is running in needs to be configured with the environment variables that are defined in the env-dev file. If you are running in a shell, you can simply source that file then start your server.js process. It has the environment variables configured correctly to work against the processes that are started if you run the docker-compose.yaml file. For example, you can to the following in a shell:

npm install
source ./env-dev
nodemon server.js

This will start up the development server and by using 'nodemon' it will auto restart as you edit the server.js file.

Now, if you make any changes to files or directories in your /my/host/directory/to/monitor, you should see fsmon pick up that change in the window where you started docker-compose, and fsmon should then push that to your server.js running in the other window which will process that event and push a message to the RabbitMQ server running in the docker-compose window. If you want to see what is going on in RabbitMQ, open your browser and go to:

RabbitMQ Console

And you can login using 'testadmin' 'testadminpass' to the RabbitMQ console and navigate to the 'test' exchange on 'testvhost' and watch the message count change if you make changes in your file directory.

The last thing you can do, is open another terminal and navigate to the 'test' directory and run:

npm install
export HOST_TEST_DIR=/my/host/directory/to/monitor
node test.js

This process will subscribe to the RabbitMQ server ('test' exhange on 'testvhost') and then will start to make changes to a test file every 10 seconds in your local directory and look for messages from RabbitMQ. This creates a loop of constantly creating, changing and removing a file in the monitored directory, which should result in a RabbitMQ message being recieved by the test script. With all this running, you can edit the server.js file and see the results immediately from the test script.

When you are finished making changes, you can kill the test process and your server by Cntl-C in each of the respective windows and then run 'docker-compose down' to shutdown the test docker environment which consists of the RabbitMQ and fsmon processes.

Checking in changes

When you are done with the change you want to make, you need to take several steps to push them out. First, depending on the type of changes made, you will need to update the version number. If they are bug fixes, you typically increment the third number in the version, if they are feature enhancements or additions that are not breaking changes for clients of this code, you can increment the second number in the version. If the code changes are substantial and/or will break client usage, you should increment the first digit in the version number.

Once you have settled on a version number:

  1. Add a line at the bottom of this file describing the changes that are associated with this new version number.
  2. Commit your changes into your local Git repository.
  3. Create a Git tag in your local repository with the version number you came up with.
  4. Push the HEAD to the remote repository.
  5. Push the tag you created to the remote repository.
  6. Next, you should generate a new Docker image that can be pushed to the remote DockerHub repository. You can build this by running the following (where x.x.x is the version number you generated in step one.):

    docker build -t mbari/fsmon-handler:x.x.x .

  7. Once the image is done, push it to DockerHub using:

    docker push mbari/fsmon-handler:x.x.x

where x.x.x is the version number you generated in step one.

Version notes

v0.0.1 - Initial version tagging

v0.0.2 - Refactored the codebase to include a docker-compose.yaml file (and supporting files) and test/test.js script to help set up environment for development and testing. Updated the README with instruction on how to use this environment in development and how to use the Docker image in deployment.

v0.0.3 - Updated to reflect changes in Slack client API and libraries.